[PATCH 0/2] Fix for multi-process gdbstub breakpoints

[PATCH 0/2] Fix for multi-process gdbstub breakpoints

[Roque Arcudia Hernandez]

This patch series modifies the gdbstub to address a bug running a
multi cluster machine in QEMU using TCG. The machine where the
problem was seen had several clusters of CPUs with similar
architectures and similar memory layout all working with physical
addresses. It was discovered under gdb debugging that a breakpoint
targeting one cluster misfired on the wrong cluster quite frequently
with no possible workaround since gdb was also unaware of any
breakpoint in that cluster and simply reported SIGTRAP.

The sequence that discovered the bug adds N inferiors and adds a
breakpoint on inferior N. Then after continuing an unrelated thread
stops the execution when its PC hits the same address of the break
targeting a different inferior.

target extended-remote :1234
add-inferior
inferior 2
attach 2
...
add-inferior
inferior N
attach N
add-symbol-file /path/to/foo.elf
break foo
> Breakpoint 1 at 0xf00add
info break
> Num     Type           Disp Enb Address    What
> 1       breakpoint     keep y   0x00f00add in foo
>                                            at foo.c:1234 inf N
continue
> Continuing.
>
> Thread 2.1 received signal SIGTRAP, Trace/breakpoint trap.
> [Switching to Thread 2.2]
> 0xf00add in ?? ()

The main problem is the current implementation of
'tcg_insert_breakpoint' and 'tcg_remove_breakpoint' insert and remove
breakpoints to all the CPUs in the system regardless of what the
remote gdb protocol implements.

If we look at the current source code of GDB we can see that the
function 'remote_target::insert_breakpoint' in file gdb/remote.c has
the intention to select the process ID of the inferior where the
break was inserted.

int
remote_target::insert_breakpoint (struct gdbarch *gdbarch,
                                  struct bp_target_info *bp_tgt)
{
...
  /* Make sure the remote is pointing at the right process, if
     necessary.  */
  if (!gdbarch_has_global_breakpoints (current_inferior ()->arch ()))
    set_general_process ();
...
}

https:sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/remote.c;h=2c3988cb5075655e8a799d1cc5d4760ad8ed426e;hb=HEAD#l11023

This would not happen when we input the 'break' in gdb but it is
deferred until the time we execute the 'continue' command. Because we
might be currently selecting an inferior that is not the one where we
previously set the breakpoint, the remote gdb has to make sure we
move the focus to the process ID of the inferior where we inserted
the break.

In the previous example this will translate to something like:

HgpN.M
Z0,00f00add,4

Another thing that is wrong with the current implementation (and it
affects both TCG and KVM mode) is that the remote gdb protocol uses
'Hg' and not 'Hc' to select the process. Functions
'gdb_breakpoint_insert' and 'gdb_breakpoint_remove' receive wrongly
'gdbserver_state.c_cpu' instead of 'gdbserver_state.g_cpu'.

This is supported by the documentation of 'H op thread-id' where op =
'c' is reserved to the step and continue:

https:sourceware.org/gdb/current/onlinedocs/gdb.html/Packets.html

And it can be verified that the function 'set_general_process' in the
previous code snippet will eventually call
'remote_target::set_general_thread' and not
'remote_target::set_continue_thread' if it needs to change focus.

A third scenario that has to be taken into account is the case of a
break on a specific thread, for instance the sequence:

inferior 1
break bar thread 1.3
break bar thread 1.4

The remote protocol expects the gdbstub to apply the break to the
process ID of inferior 1 and considers the specific thread-id as a
breakpoint condition (not too different from a 'break if').

In this case the packet exchange may look like:

Hgp1.1
Z0,00ba2add,4

There wouldn't be an independent set of packets for 'Hgp1.3' and
'Hgp1.4'.

In the gdb source code, the handling of the specific thread-id
happens during breakpoint evaluation in function
'bpstat_check_breakpoint_conditions' of file gdb/breakpoint.c

https:sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/breakpoint.c;h=17bd627f867cf3d4dc81322ed1919ba40cbb237d;hb=HEAD#l5550

The proposed fix inserts or removes a breakpoint to all the cpus
sharing the same process ID as the one selected with the latest
received 'Hg' packet.

Roque Arcudia Hernandez (2):
  gdbstub: Fix wrong CPUState pointer in breakpoint functions
  gdbstub: Apply breakpoints only to the selected PID

 accel/tcg/tcg-accel-ops.c | 15 +++++++++++++++
 gdbstub/gdbstub.c         | 11 ++++++++---
 include/exec/gdbstub.h    | 15 +++++++++++++++
 3 files changed, 38 insertions(+), 3 deletions(-)

-- 
2.46.0.469.g59c65b2a67-goog

[PATCH 1/2] gdbstub: Fix wrong CPUState pointer in breakpoint functions

[Roque Arcudia Hernandez]

In the context of using the remote gdb with multiple
processes/inferiors (multiple cluster machine) a given breakpoint
will target an specific inferior. If needed the remote protocol will
use the packet 'H op thread-id' with op = 'g' to change focus to the
inferior we want to insert/remove the breakpoint to, for instance
'Hgp3.3' and not 'Hcp3.3'.

This is supported by the documentation of the H packets:

 > 'H op thread-id'
 > Set thread for subsequent operations ('m', 'M', 'g', 'G',
 > et.al.). Depending on the operation to be performed, op should be
 > 'c' for step and continue operations (note that this is
 > deprecated, supporting the 'vCont' command is a better option),
 > and 'g' for other operations.

This can also be verified in the GDB source code file gdb/remote.c.
Functions remote_target::insert_breakpoint and
remote_target::remove_breakpoint will eventually call
remote_target::set_general_thread if it needs to change the process
focus and not remote_target::set_continue_thread.

This can be seen around a comment that says:

      /* Make sure the remote is pointing at the right process, if
         necessary.  */

Google-Bug-Id: 355027002
Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
---
 gdbstub/gdbstub.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
index d08568cea0..98574eba68 100644
--- a/gdbstub/gdbstub.c
+++ b/gdbstub/gdbstub.c
@@ -1148,7 +1148,7 @@ static void handle_insert_bp(GArray *params, void *user_ctx)
         return;
     }
 
-    res = gdb_breakpoint_insert(gdbserver_state.c_cpu,
+    res = gdb_breakpoint_insert(gdbserver_state.g_cpu,
                                 gdb_get_cmd_param(params, 0)->val_ul,
                                 gdb_get_cmd_param(params, 1)->val_ull,
                                 gdb_get_cmd_param(params, 2)->val_ull);
@@ -1172,7 +1172,7 @@ static void handle_remove_bp(GArray *params, void *user_ctx)
         return;
     }
 
-    res = gdb_breakpoint_remove(gdbserver_state.c_cpu,
+    res = gdb_breakpoint_remove(gdbserver_state.g_cpu,
                                 gdb_get_cmd_param(params, 0)->val_ul,
                                 gdb_get_cmd_param(params, 1)->val_ull,
                                 gdb_get_cmd_param(params, 2)->val_ull);
-- 
2.46.0.469.g59c65b2a67-goog

Re: [PATCH 1/2] gdbstub: Fix wrong CPUState pointer in breakpoint functions

[Alex Bennée]

Roque Arcudia Hernandez <roqueh@google.com> writes:

> In the context of using the remote gdb with multiple
> processes/inferiors (multiple cluster machine) a given breakpoint
> will target an specific inferior. If needed the remote protocol will
> use the packet 'H op thread-id' with op = 'g' to change focus to the
> inferior we want to insert/remove the breakpoint to, for instance
> 'Hgp3.3' and not 'Hcp3.3'.
>
> This is supported by the documentation of the H packets:
>
>  > 'H op thread-id'
>  > Set thread for subsequent operations ('m', 'M', 'g', 'G',
>  > et.al.). Depending on the operation to be performed, op should be
>  > 'c' for step and continue operations (note that this is
>  > deprecated, supporting the 'vCont' command is a better option),
>  > and 'g' for other operations.

Can we better comment:

    CPUState *c_cpu; /* current CPU for step/continue ops */
    CPUState *g_cpu; /* current CPU for other ops */

in GDBState?

>
> This can also be verified in the GDB source code file gdb/remote.c.
> Functions remote_target::insert_breakpoint and
> remote_target::remove_breakpoint will eventually call
> remote_target::set_general_thread if it needs to change the process
> focus and not remote_target::set_continue_thread.
>
> This can be seen around a comment that says:
>
>       /* Make sure the remote is pointing at the right process, if
>          necessary.  */
>
> Google-Bug-Id: 355027002
> Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> ---
>  gdbstub/gdbstub.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
> index d08568cea0..98574eba68 100644
> --- a/gdbstub/gdbstub.c
> +++ b/gdbstub/gdbstub.c
> @@ -1148,7 +1148,7 @@ static void handle_insert_bp(GArray *params, void *user_ctx)
>          return;
>      }
>  
> -    res = gdb_breakpoint_insert(gdbserver_state.c_cpu,
> +    res = gdb_breakpoint_insert(gdbserver_state.g_cpu,
>                                  gdb_get_cmd_param(params, 0)->val_ul,
>                                  gdb_get_cmd_param(params, 1)->val_ull,
>                                  gdb_get_cmd_param(params, 2)->val_ull);
> @@ -1172,7 +1172,7 @@ static void handle_remove_bp(GArray *params, void *user_ctx)
>          return;
>      }
>  
> -    res = gdb_breakpoint_remove(gdbserver_state.c_cpu,
> +    res = gdb_breakpoint_remove(gdbserver_state.g_cpu,
>                                  gdb_get_cmd_param(params, 0)->val_ul,
>                                  gdb_get_cmd_param(params, 1)->val_ull,
>                                  gdb_get_cmd_param(params, 2)->val_ull);

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 1/2] gdbstub: Fix wrong CPUState pointer in breakpoint functions

[Roque Arcudia Hernandez]

I'm adding extra documentation to those fields in a new patchset.

On Fri, Oct 4, 2024 at 1:46 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Roque Arcudia Hernandez <roqueh@google.com> writes:
>
> > In the context of using the remote gdb with multiple
> > processes/inferiors (multiple cluster machine) a given breakpoint
> > will target an specific inferior. If needed the remote protocol will
> > use the packet 'H op thread-id' with op = 'g' to change focus to the
> > inferior we want to insert/remove the breakpoint to, for instance
> > 'Hgp3.3' and not 'Hcp3.3'.
> >
> > This is supported by the documentation of the H packets:
> >
> >  > 'H op thread-id'
> >  > Set thread for subsequent operations ('m', 'M', 'g', 'G',
> >  > et.al.). Depending on the operation to be performed, op should be
> >  > 'c' for step and continue operations (note that this is
> >  > deprecated, supporting the 'vCont' command is a better option),
> >  > and 'g' for other operations.
>
> Can we better comment:
>
>     CPUState *c_cpu; /* current CPU for step/continue ops */
>     CPUState *g_cpu; /* current CPU for other ops */
>
> in GDBState?
>
> >
> > This can also be verified in the GDB source code file gdb/remote.c.
> > Functions remote_target::insert_breakpoint and
> > remote_target::remove_breakpoint will eventually call
> > remote_target::set_general_thread if it needs to change the process
> > focus and not remote_target::set_continue_thread.
> >
> > This can be seen around a comment that says:
> >
> >       /* Make sure the remote is pointing at the right process, if
> >          necessary.  */
> >
> > Google-Bug-Id: 355027002
> > Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> > ---
> >  gdbstub/gdbstub.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
> > index d08568cea0..98574eba68 100644
> > --- a/gdbstub/gdbstub.c
> > +++ b/gdbstub/gdbstub.c
> > @@ -1148,7 +1148,7 @@ static void handle_insert_bp(GArray *params, void *user_ctx)
> >          return;
> >      }
> >
> > -    res = gdb_breakpoint_insert(gdbserver_state.c_cpu,
> > +    res = gdb_breakpoint_insert(gdbserver_state.g_cpu,
> >                                  gdb_get_cmd_param(params, 0)->val_ul,
> >                                  gdb_get_cmd_param(params, 1)->val_ull,
> >                                  gdb_get_cmd_param(params, 2)->val_ull);
> > @@ -1172,7 +1172,7 @@ static void handle_remove_bp(GArray *params, void *user_ctx)
> >          return;
> >      }
> >
> > -    res = gdb_breakpoint_remove(gdbserver_state.c_cpu,
> > +    res = gdb_breakpoint_remove(gdbserver_state.g_cpu,
> >                                  gdb_get_cmd_param(params, 0)->val_ul,
> >                                  gdb_get_cmd_param(params, 1)->val_ull,
> >                                  gdb_get_cmd_param(params, 2)->val_ull);
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

Re: [PATCH 1/2] gdbstub: Fix wrong CPUState pointer in breakpoint functions

[Philippe Mathieu-Daudé]

Hi Roque,

On 6/9/24 19:54, Roque Arcudia Hernandez wrote:
> In the context of using the remote gdb with multiple
> processes/inferiors (multiple cluster machine) a given breakpoint
> will target an specific inferior. If needed the remote protocol will
> use the packet 'H op thread-id' with op = 'g' to change focus to the
> inferior we want to insert/remove the breakpoint to, for instance
> 'Hgp3.3' and not 'Hcp3.3'.
> 
> This is supported by the documentation of the H packets:
> 
>   > 'H op thread-id'
>   > Set thread for subsequent operations ('m', 'M', 'g', 'G',
>   > et.al.). Depending on the operation to be performed, op should be
>   > 'c' for step and continue operations (note that this is
>   > deprecated, supporting the 'vCont' command is a better option),
>   > and 'g' for other operations.
> 
> This can also be verified in the GDB source code file gdb/remote.c.
> Functions remote_target::insert_breakpoint and
> remote_target::remove_breakpoint will eventually call
> remote_target::set_general_thread if it needs to change the process
> focus and not remote_target::set_continue_thread.
> 
> This can be seen around a comment that says:
> 
>        /* Make sure the remote is pointing at the right process, if
>           necessary.  */
> 
> Google-Bug-Id: 355027002

Where can we find more information on this bug ID?
I tried various query in the Google public tracker but
couldn't find anything.
(i.e. https://issuetracker.google.com/issues?q=canonicalid:355027002)

> Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> ---
>   gdbstub/gdbstub.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)

Re: [PATCH 1/2] gdbstub: Fix wrong CPUState pointer in breakpoint functions

[Roque Arcudia Hernandez]

Hello Philippe,

The Google-Bug-Id is an internal ID and should have been removed. The
information in the bug was mostly transcribed in the cover letter of
this patch series.

Thanks

Roque

On Mon, Oct 7, 2024 at 1:06 PM Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> Hi Roque,
>
> On 6/9/24 19:54, Roque Arcudia Hernandez wrote:
> > In the context of using the remote gdb with multiple
> > processes/inferiors (multiple cluster machine) a given breakpoint
> > will target an specific inferior. If needed the remote protocol will
> > use the packet 'H op thread-id' with op = 'g' to change focus to the
> > inferior we want to insert/remove the breakpoint to, for instance
> > 'Hgp3.3' and not 'Hcp3.3'.
> >
> > This is supported by the documentation of the H packets:
> >
> >   > 'H op thread-id'
> >   > Set thread for subsequent operations ('m', 'M', 'g', 'G',
> >   > et.al.). Depending on the operation to be performed, op should be
> >   > 'c' for step and continue operations (note that this is
> >   > deprecated, supporting the 'vCont' command is a better option),
> >   > and 'g' for other operations.
> >
> > This can also be verified in the GDB source code file gdb/remote.c.
> > Functions remote_target::insert_breakpoint and
> > remote_target::remove_breakpoint will eventually call
> > remote_target::set_general_thread if it needs to change the process
> > focus and not remote_target::set_continue_thread.
> >
> > This can be seen around a comment that says:
> >
> >        /* Make sure the remote is pointing at the right process, if
> >           necessary.  */
> >
> > Google-Bug-Id: 355027002
>
> Where can we find more information on this bug ID?
> I tried various query in the Google public tracker but
> couldn't find anything.
> (i.e. https://issuetracker.google.com/issues?q=canonicalid:355027002)
>
> > Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> > ---
> >   gdbstub/gdbstub.c | 4 ++--
> >   1 file changed, 2 insertions(+), 2 deletions(-)
>

[PATCH 2/2] gdbstub: Apply breakpoints only to the selected PID

[Roque Arcudia Hernandez]

In the context of using the remote gdb with multiple
processes/inferiors (multi cluster machine) a given breakpoint will
target an specific inferior. Current implementation of
tcg_insert_breakpoint and tcg_remove_breakpoint apply a given
breakpoint to all the cpus available in the system.

This is not how gdb expects the remote protocol to behave. If we
refer to the current source of gdb, in the function
remote_target::insert_breakpoint in the file gdb/remote.c we can see
the intention to have the right process selected before breakpoint
insertion:

int
remote_target::insert_breakpoint (struct gdbarch *gdbarch,
                                  struct bp_target_info *bp_tgt)
{
...
  /* Make sure the remote is pointing at the right process, if
     necessary.  */
  if (!gdbarch_has_global_breakpoints (current_inferior ()->arch ()))
    set_general_process ();
...
}

Since most platforms do not have global breakpoints, this typically
will result in an 'Hg' Packet sent before 'z/Z', if gdb is not
pointing to the desired process.

For instance this is a concrete example obtained with a trace dump:

gdbstub_io_command Received: Hgp2.2
gdbstub_io_command Received: Z0,4000ded0,4

Only the CPUs associated with the selected process ID should insert
or remove the breakpoint. It is important to apply it to all the CPUs
in the process ID regardless of the particular thread selected by the
'Hg' packet because even in the case of a thread specific breakpoint.
A breakpoint on a specific thread is treated as a conditional break
similar to a 'break if'. This can be read in the code and comments of
function bpstat_check_breakpoint_conditions in the file
gdb/breakpoint.c

/* For breakpoints that are currently marked as telling gdb to stop,
   check conditions (condition proper, frame, thread and ignore count)
   of breakpoint referred to by BS.  If we should not stop for this
   breakpoint, set BS->stop to 0.  */

static void
bpstat_check_breakpoint_conditions (bpstat *bs, thread_info *thread)

The patch needs to expose the currently private function
gdb_get_cpu_pid to the TCG and also expose the value of
gdbserver_state.multiprocess. The PID filtering will only be
applicable to multiprocess gdb because the PIDs are only defined in
that context.

Google-Bug-Id: 355027002
Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
---
 accel/tcg/tcg-accel-ops.c | 15 +++++++++++++++
 gdbstub/gdbstub.c         |  7 ++++++-
 include/exec/gdbstub.h    | 15 +++++++++++++++
 3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
index 3c19e68a79..557091c15e 100644
--- a/accel/tcg/tcg-accel-ops.c
+++ b/accel/tcg/tcg-accel-ops.c
@@ -33,6 +33,7 @@
 #include "qemu/guest-random.h"
 #include "qemu/timer.h"
 #include "exec/exec-all.h"
+#include "exec/gdbstub.h"
 #include "exec/hwaddr.h"
 #include "exec/tb-flush.h"
 #include "gdbstub/enums.h"
@@ -132,11 +133,15 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
 {
     CPUState *cpu;
     int err = 0;
+    bool match_pid = gdb_is_multiprocess();
 
     switch (type) {
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
         CPU_FOREACH(cpu) {
+            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
+                continue;
+            }
             err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
             if (err) {
                 break;
@@ -147,6 +152,9 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
         CPU_FOREACH(cpu) {
+            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
+                continue;
+            }
             err = cpu_watchpoint_insert(cpu, addr, len,
                                         xlat_gdb_type(cpu, type), NULL);
             if (err) {
@@ -163,11 +171,15 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
 {
     CPUState *cpu;
     int err = 0;
+    bool match_pid = gdb_is_multiprocess();
 
     switch (type) {
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
         CPU_FOREACH(cpu) {
+            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
+                continue;
+            }
             err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
             if (err) {
                 break;
@@ -178,6 +190,9 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
         CPU_FOREACH(cpu) {
+            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
+                continue;
+            }
             err = cpu_watchpoint_remove(cpu, addr, len,
                                         xlat_gdb_type(cpu, type));
             if (err) {
diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
index 98574eba68..628e599692 100644
--- a/gdbstub/gdbstub.c
+++ b/gdbstub/gdbstub.c
@@ -199,7 +199,12 @@ void gdb_memtox(GString *buf, const char *mem, int len)
     }
 }
 
-static uint32_t gdb_get_cpu_pid(CPUState *cpu)
+bool gdb_is_multiprocess(void)
+{
+    return gdbserver_state.multiprocess;
+}
+
+uint32_t gdb_get_cpu_pid(CPUState *cpu)
 {
 #ifdef CONFIG_USER_ONLY
     return getpid();
diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
index d73f424f56..a7c95d215b 100644
--- a/include/exec/gdbstub.h
+++ b/include/exec/gdbstub.h
@@ -138,6 +138,21 @@ GArray *gdb_get_register_list(CPUState *cpu);
 
 void gdb_set_stop_cpu(CPUState *cpu);
 
+/**
+ * gdb_get_cpu_pid() - Get the PID assigned to a CPU
+ * @cpu - the CPU to query
+ *
+ * Return: The PID associated with the cpu
+ */
+uint32_t gdb_get_cpu_pid(CPUState *cpu);
+
+/**
+ * gdb_is_multiprocess() - Tells if the gdb server supports multiple processes
+ *
+ * Return: True if supported
+ */
+bool gdb_is_multiprocess(void);
+
 /* in gdbstub-xml.c, generated by scripts/feature_to_c.py */
 extern const GDBFeature gdb_static_features[];
 
-- 
2.46.0.469.g59c65b2a67-goog

Re: [PATCH 2/2] gdbstub: Apply breakpoints only to the selected PID

[Alex Bennée]

Roque Arcudia Hernandez <roqueh@google.com> writes:

> In the context of using the remote gdb with multiple
> processes/inferiors (multi cluster machine) a given breakpoint will
> target an specific inferior. Current implementation of
> tcg_insert_breakpoint and tcg_remove_breakpoint apply a given
> breakpoint to all the cpus available in the system.

OK I can see this needs fixing.

> Only the CPUs associated with the selected process ID should insert
> or remove the breakpoint. It is important to apply it to all the CPUs
> in the process ID regardless of the particular thread selected by the
> 'Hg' packet because even in the case of a thread specific breakpoint.
> A breakpoint on a specific thread is treated as a conditional break
> similar to a 'break if'. This can be read in the code and comments of
> function bpstat_check_breakpoint_conditions in the file
> gdb/breakpoint.c
>
> /* For breakpoints that are currently marked as telling gdb to stop,
>    check conditions (condition proper, frame, thread and ignore count)
>    of breakpoint referred to by BS.  If we should not stop for this
>    breakpoint, set BS->stop to 0.  */
>
> static void
> bpstat_check_breakpoint_conditions (bpstat *bs, thread_info *thread)
>
> The patch needs to expose the currently private function
> gdb_get_cpu_pid to the TCG and also expose the value of
> gdbserver_state.multiprocess. The PID filtering will only be
> applicable to multiprocess gdb because the PIDs are only defined in
> that context.

I don't like exposing those private functions, especially as the PID
terminology is confusing. Could we not keep all the logic in the gdbstub
itself so the tests become something like:

    case GDB_BREAKPOINT_SW:
    case GDB_BREAKPOINT_HW:
        CPU_FOREACH(cpu) {
            if (gdb_cpu_in_source_group(cs, cpu)) {
                err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
                if (err) {
                    break;
                }
            }
        }
        return err;

?

Ilya has also posted a large series for non-stop mode on *-user guests
which is on my review queue which I would want to check doesn't conflict
with this:

  Message-ID: <20240923162208.90745-1-iii@linux.ibm.com>
  Date: Mon, 23 Sep 2024 18:12:55 +0200
  Subject: [PATCH 00/18] Stop all qemu-cpu threads on a breakpoint
  From: Ilya Leoshkevich <iii@linux.ibm.com>

>
> Google-Bug-Id: 355027002
> Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> ---
>  accel/tcg/tcg-accel-ops.c | 15 +++++++++++++++
>  gdbstub/gdbstub.c         |  7 ++++++-
>  include/exec/gdbstub.h    | 15 +++++++++++++++
>  3 files changed, 36 insertions(+), 1 deletion(-)
>
> diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
> index 3c19e68a79..557091c15e 100644
> --- a/accel/tcg/tcg-accel-ops.c
> +++ b/accel/tcg/tcg-accel-ops.c
> @@ -33,6 +33,7 @@
>  #include "qemu/guest-random.h"
>  #include "qemu/timer.h"
>  #include "exec/exec-all.h"
> +#include "exec/gdbstub.h"
>  #include "exec/hwaddr.h"
>  #include "exec/tb-flush.h"
>  #include "gdbstub/enums.h"
> @@ -132,11 +133,15 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
>  {
>      CPUState *cpu;
>      int err = 0;
> +    bool match_pid = gdb_is_multiprocess();
>  
>      switch (type) {
>      case GDB_BREAKPOINT_SW:
>      case GDB_BREAKPOINT_HW:
>          CPU_FOREACH(cpu) {
> +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> +                continue;
> +            }
>              err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
>              if (err) {
>                  break;
> @@ -147,6 +152,9 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
>      case GDB_WATCHPOINT_READ:
>      case GDB_WATCHPOINT_ACCESS:
>          CPU_FOREACH(cpu) {
> +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> +                continue;
> +            }
>              err = cpu_watchpoint_insert(cpu, addr, len,
>                                          xlat_gdb_type(cpu, type), NULL);
>              if (err) {
> @@ -163,11 +171,15 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
>  {
>      CPUState *cpu;
>      int err = 0;
> +    bool match_pid = gdb_is_multiprocess();
>  
>      switch (type) {
>      case GDB_BREAKPOINT_SW:
>      case GDB_BREAKPOINT_HW:
>          CPU_FOREACH(cpu) {
> +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> +                continue;
> +            }
>              err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
>              if (err) {
>                  break;
> @@ -178,6 +190,9 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
>      case GDB_WATCHPOINT_READ:
>      case GDB_WATCHPOINT_ACCESS:
>          CPU_FOREACH(cpu) {
> +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> +                continue;
> +            }
>              err = cpu_watchpoint_remove(cpu, addr, len,
>                                          xlat_gdb_type(cpu, type));
>              if (err) {
> diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
> index 98574eba68..628e599692 100644
> --- a/gdbstub/gdbstub.c
> +++ b/gdbstub/gdbstub.c
> @@ -199,7 +199,12 @@ void gdb_memtox(GString *buf, const char *mem, int len)
>      }
>  }
>  
> -static uint32_t gdb_get_cpu_pid(CPUState *cpu)
> +bool gdb_is_multiprocess(void)
> +{
> +    return gdbserver_state.multiprocess;
> +}
> +
> +uint32_t gdb_get_cpu_pid(CPUState *cpu)
>  {
>  #ifdef CONFIG_USER_ONLY
>      return getpid();
> diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
> index d73f424f56..a7c95d215b 100644
> --- a/include/exec/gdbstub.h
> +++ b/include/exec/gdbstub.h
> @@ -138,6 +138,21 @@ GArray *gdb_get_register_list(CPUState *cpu);
>  
>  void gdb_set_stop_cpu(CPUState *cpu);
>  
> +/**
> + * gdb_get_cpu_pid() - Get the PID assigned to a CPU
> + * @cpu - the CPU to query
> + *
> + * Return: The PID associated with the cpu
> + */
> +uint32_t gdb_get_cpu_pid(CPUState *cpu);
> +
> +/**
> + * gdb_is_multiprocess() - Tells if the gdb server supports multiple processes
> + *
> + * Return: True if supported
> + */
> +bool gdb_is_multiprocess(void);
> +
>  /* in gdbstub-xml.c, generated by scripts/feature_to_c.py */
>  extern const GDBFeature gdb_static_features[];

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 2/2] gdbstub: Apply breakpoints only to the selected PID

[Roque Arcudia Hernandez]

I'm reimplementing this in a new patchset with a new function
gdb_cpu_in_source_group instead of making public the PID and
multiptocess functions.

On Mon, Oct 7, 2024 at 3:15 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Roque Arcudia Hernandez <roqueh@google.com> writes:
>
> > In the context of using the remote gdb with multiple
> > processes/inferiors (multi cluster machine) a given breakpoint will
> > target an specific inferior. Current implementation of
> > tcg_insert_breakpoint and tcg_remove_breakpoint apply a given
> > breakpoint to all the cpus available in the system.
>
> OK I can see this needs fixing.
>
> > Only the CPUs associated with the selected process ID should insert
> > or remove the breakpoint. It is important to apply it to all the CPUs
> > in the process ID regardless of the particular thread selected by the
> > 'Hg' packet because even in the case of a thread specific breakpoint.
> > A breakpoint on a specific thread is treated as a conditional break
> > similar to a 'break if'. This can be read in the code and comments of
> > function bpstat_check_breakpoint_conditions in the file
> > gdb/breakpoint.c
> >
> > /* For breakpoints that are currently marked as telling gdb to stop,
> >    check conditions (condition proper, frame, thread and ignore count)
> >    of breakpoint referred to by BS.  If we should not stop for this
> >    breakpoint, set BS->stop to 0.  */
> >
> > static void
> > bpstat_check_breakpoint_conditions (bpstat *bs, thread_info *thread)
> >
> > The patch needs to expose the currently private function
> > gdb_get_cpu_pid to the TCG and also expose the value of
> > gdbserver_state.multiprocess. The PID filtering will only be
> > applicable to multiprocess gdb because the PIDs are only defined in
> > that context.
>
> I don't like exposing those private functions, especially as the PID
> terminology is confusing. Could we not keep all the logic in the gdbstub
> itself so the tests become something like:
>
>     case GDB_BREAKPOINT_SW:
>     case GDB_BREAKPOINT_HW:
>         CPU_FOREACH(cpu) {
>             if (gdb_cpu_in_source_group(cs, cpu)) {
>                 err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
>                 if (err) {
>                     break;
>                 }
>             }
>         }
>         return err;
>
> ?
>
> Ilya has also posted a large series for non-stop mode on *-user guests
> which is on my review queue which I would want to check doesn't conflict
> with this:
>
>   Message-ID: <20240923162208.90745-1-iii@linux.ibm.com>
>   Date: Mon, 23 Sep 2024 18:12:55 +0200
>   Subject: [PATCH 00/18] Stop all qemu-cpu threads on a breakpoint
>   From: Ilya Leoshkevich <iii@linux.ibm.com>
>
> >
> > Google-Bug-Id: 355027002
> > Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
> > ---
> >  accel/tcg/tcg-accel-ops.c | 15 +++++++++++++++
> >  gdbstub/gdbstub.c         |  7 ++++++-
> >  include/exec/gdbstub.h    | 15 +++++++++++++++
> >  3 files changed, 36 insertions(+), 1 deletion(-)
> >
> > diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
> > index 3c19e68a79..557091c15e 100644
> > --- a/accel/tcg/tcg-accel-ops.c
> > +++ b/accel/tcg/tcg-accel-ops.c
> > @@ -33,6 +33,7 @@
> >  #include "qemu/guest-random.h"
> >  #include "qemu/timer.h"
> >  #include "exec/exec-all.h"
> > +#include "exec/gdbstub.h"
> >  #include "exec/hwaddr.h"
> >  #include "exec/tb-flush.h"
> >  #include "gdbstub/enums.h"
> > @@ -132,11 +133,15 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
> >  {
> >      CPUState *cpu;
> >      int err = 0;
> > +    bool match_pid = gdb_is_multiprocess();
> >
> >      switch (type) {
> >      case GDB_BREAKPOINT_SW:
> >      case GDB_BREAKPOINT_HW:
> >          CPU_FOREACH(cpu) {
> > +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> > +                continue;
> > +            }
> >              err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
> >              if (err) {
> >                  break;
> > @@ -147,6 +152,9 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
> >      case GDB_WATCHPOINT_READ:
> >      case GDB_WATCHPOINT_ACCESS:
> >          CPU_FOREACH(cpu) {
> > +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> > +                continue;
> > +            }
> >              err = cpu_watchpoint_insert(cpu, addr, len,
> >                                          xlat_gdb_type(cpu, type), NULL);
> >              if (err) {
> > @@ -163,11 +171,15 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
> >  {
> >      CPUState *cpu;
> >      int err = 0;
> > +    bool match_pid = gdb_is_multiprocess();
> >
> >      switch (type) {
> >      case GDB_BREAKPOINT_SW:
> >      case GDB_BREAKPOINT_HW:
> >          CPU_FOREACH(cpu) {
> > +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> > +                continue;
> > +            }
> >              err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
> >              if (err) {
> >                  break;
> > @@ -178,6 +190,9 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
> >      case GDB_WATCHPOINT_READ:
> >      case GDB_WATCHPOINT_ACCESS:
> >          CPU_FOREACH(cpu) {
> > +            if (match_pid && gdb_get_cpu_pid(cpu) != gdb_get_cpu_pid(cs)) {
> > +                continue;
> > +            }
> >              err = cpu_watchpoint_remove(cpu, addr, len,
> >                                          xlat_gdb_type(cpu, type));
> >              if (err) {
> > diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
> > index 98574eba68..628e599692 100644
> > --- a/gdbstub/gdbstub.c
> > +++ b/gdbstub/gdbstub.c
> > @@ -199,7 +199,12 @@ void gdb_memtox(GString *buf, const char *mem, int len)
> >      }
> >  }
> >
> > -static uint32_t gdb_get_cpu_pid(CPUState *cpu)
> > +bool gdb_is_multiprocess(void)
> > +{
> > +    return gdbserver_state.multiprocess;
> > +}
> > +
> > +uint32_t gdb_get_cpu_pid(CPUState *cpu)
> >  {
> >  #ifdef CONFIG_USER_ONLY
> >      return getpid();
> > diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
> > index d73f424f56..a7c95d215b 100644
> > --- a/include/exec/gdbstub.h
> > +++ b/include/exec/gdbstub.h
> > @@ -138,6 +138,21 @@ GArray *gdb_get_register_list(CPUState *cpu);
> >
> >  void gdb_set_stop_cpu(CPUState *cpu);
> >
> > +/**
> > + * gdb_get_cpu_pid() - Get the PID assigned to a CPU
> > + * @cpu - the CPU to query
> > + *
> > + * Return: The PID associated with the cpu
> > + */
> > +uint32_t gdb_get_cpu_pid(CPUState *cpu);
> > +
> > +/**
> > + * gdb_is_multiprocess() - Tells if the gdb server supports multiple processes
> > + *
> > + * Return: True if supported
> > + */
> > +bool gdb_is_multiprocess(void);
> > +
> >  /* in gdbstub-xml.c, generated by scripts/feature_to_c.py */
> >  extern const GDBFeature gdb_static_features[];
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

[PATCH 0/2] Fix for multi-process gdbstub breakpoints

[Nabih Estefan]

This patch series modifies the gdbstub to address a bug running a
multi cluster machine in QEMU using TCG. The machine where the
problem was seen had several clusters of CPUs with similar
architectures and similar memory layout all working with physical
addresses. It was discovered under gdb debugging that a breakpoint
targeting one cluster misfired on the wrong cluster quite frequently
with no possible workaround since gdb was also unaware of any
breakpoint in that cluster and simply reported SIGTRAP.

The sequence that discovered the bug adds N inferiors and adds a
breakpoint on inferior N. Then after continuing an unrelated thread
stops the execution when its PC hits the same address of the break
targeting a different inferior.

target extended-remote :1234
add-inferior
inferior 2
attach 2
...
add-inferior
inferior N
attach N
add-symbol-file /path/to/foo.elf
break foo
> Breakpoint 1 at 0xf00add
info break
> Num     Type           Disp Enb Address    What
> 1       breakpoint     keep y   0x00f00add in foo
>                                            at foo.c:1234 inf N
continue
> Continuing.
>
> Thread 2.1 received signal SIGTRAP, Trace/breakpoint trap.
> [Switching to Thread 2.2]
> 0xf00add in ?? ()

The main problem is the current implementation of
'tcg_insert_breakpoint' and 'tcg_remove_breakpoint' insert and remove
breakpoints to all the CPUs in the system regardless of what the
remote gdb protocol implements.

If we look at the current source code of GDB we can see that the
function 'remote_target::insert_breakpoint' in file gdb/remote.c has
the intention to select the process ID of the inferior where the
break was inserted.

int
remote_target::insert_breakpoint (struct gdbarch *gdbarch,
                                  struct bp_target_info *bp_tgt)
{
...
  /* Make sure the remote is pointing at the right process, if
     necessary.  */
  if (!gdbarch_has_global_breakpoints (current_inferior ()->arch ()))
    set_general_process ();
...
}

https:sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/remote.c;h=2c3988cb5075655e8a799d1cc5d4760ad8ed426e;hb=HEAD#l11023

This would not happen when we input the 'break' in gdb but it is
deferred until the time we execute the 'continue' command. Because we
might be currently selecting an inferior that is not the one where we
previously set the breakpoint, the remote gdb has to make sure we
move the focus to the process ID of the inferior where we inserted
the break.

In the previous example this will translate to something like:

HgpN.M
Z0,00f00add,4

Another thing that is wrong with the current implementation (and it
affects both TCG and KVM mode) is that the remote gdb protocol uses
'Hg' and not 'Hc' to select the process. Functions
'gdb_breakpoint_insert' and 'gdb_breakpoint_remove' receive wrongly
'gdbserver_state.c_cpu' instead of 'gdbserver_state.g_cpu'.

This is supported by the documentation of 'H op thread-id' where op =
'c' is reserved to the step and continue:

https:sourceware.org/gdb/current/onlinedocs/gdb.html/Packets.html

And it can be verified that the function 'set_general_process' in the
previous code snippet will eventually call
'remote_target::set_general_thread' and not
'remote_target::set_continue_thread' if it needs to change focus.

A third scenario that has to be taken into account is the case of a
break on a specific thread, for instance the sequence:

inferior 1
break bar thread 1.3
break bar thread 1.4

The remote protocol expects the gdbstub to apply the break to the
process ID of inferior 1 and considers the specific thread-id as a
breakpoint condition (not too different from a 'break if').

In this case the packet exchange may look like:

Hgp1.1
Z0,00ba2add,4

There wouldn't be an independent set of packets for 'Hgp1.3' and
'Hgp1.4'.

In the gdb source code, the handling of the specific thread-id
happens during breakpoint evaluation in function
'bpstat_check_breakpoint_conditions' of file gdb/breakpoint.c

https:sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/breakpoint.c;h=17bd627f867cf3d4dc81322ed1919ba40cbb237d;hb=HEAD#l5550

The proposed fix inserts or removes a breakpoint to all the cpus
sharing the same process ID as the one selected with the latest
received 'Hg' packet.

Roque Arcudia Hernandez (2):
  gdbstub: Fix wrong CPUState pointer in breakpoint functions
  gdbstub: Apply breakpoints only to the selected PID

 accel/tcg/tcg-accel-ops.c | 37 +++++++++++++++++++++++--------------
 gdbstub/gdbstub.c         | 10 ++++++++--
 gdbstub/internals.h       | 13 +++++++++++--
 include/exec/gdbstub.h    | 12 ++++++++++++
 4 files changed, 54 insertions(+), 18 deletions(-)

-- 
2.49.0.1015.ga840276032-goog

[PATCH 2/2] gdbstub: Apply breakpoints only to the selected PID

[Nabih Estefan]

From: Roque Arcudia Hernandez <roqueh@google.com>

In the context of using the remote gdb with multiple
processes/inferiors (multi cluster machine) a given breakpoint will
target an specific inferior. Current implementation of
tcg_insert_breakpoint and tcg_remove_breakpoint apply a given
breakpoint to all the cpus available in the system.

This is not how gdb expects the remote protocol to behave. If we
refer to the current source of gdb, in the function
remote_target::insert_breakpoint in the file gdb/remote.c we can see
the intention to have the right process selected before breakpoint
insertion:

int
remote_target::insert_breakpoint (struct gdbarch *gdbarch,
                                  struct bp_target_info *bp_tgt)
{
...
  /* Make sure the remote is pointing at the right process, if
     necessary.  */
  if (!gdbarch_has_global_breakpoints (current_inferior ()->arch ()))
    set_general_process ();
...
}

Since most platforms do not have global breakpoints, this typically
will result in an 'Hg' Packet sent before 'z/Z', if gdb is not
pointing to the desired process.

For instance this is a concrete example obtained with a trace dump:

gdbstub_io_command Received: Hgp2.2
gdbstub_io_command Received: Z0,4000ded0,4

Only the CPUs associated with the selected process ID should insert
or remove the breakpoint. It is important to apply it to all the CPUs
in the process ID regardless of the particular thread selected by the
'Hg' packet because even in the case of a thread specific breakpoint.
A breakpoint on a specific thread is treated as a conditional break
similar to a 'break if'. This can be read in the code and comments of
function bpstat_check_breakpoint_conditions in the file
gdb/breakpoint.c

/* For breakpoints that are currently marked as telling gdb to stop,
   check conditions (condition proper, frame, thread and ignore count)
   of breakpoint referred to by BS.  If we should not stop for this
   breakpoint, set BS->stop to 0.  */

static void
bpstat_check_breakpoint_conditions (bpstat *bs, thread_info *thread)

The patch needs to expose the currently private function
gdb_get_cpu_pid to the TCG and also expose the value of
gdbserver_state.multiprocess. The PID filtering will only be
applicable to multiprocess gdb because the PIDs are only defined in
that context.

Signed-off-by: Roque Arcudia Hernandez <roqueh@google.com>
Signed-off-by: Nabih Estefan <nabihestefan@google.com>
---
 accel/tcg/tcg-accel-ops.c | 37 +++++++++++++++++++++++--------------
 gdbstub/gdbstub.c         |  6 ++++++
 include/exec/gdbstub.h    | 12 ++++++++++++
 3 files changed, 41 insertions(+), 14 deletions(-)

diff --git a/accel/tcg/tcg-accel-ops.c b/accel/tcg/tcg-accel-ops.c
index b24d6a7562..aca476cdf5 100644
--- a/accel/tcg/tcg-accel-ops.c
+++ b/accel/tcg/tcg-accel-ops.c
@@ -34,6 +34,7 @@
 #include "qemu/guest-random.h"
 #include "qemu/timer.h"
 #include "exec/cputlb.h"
+#include "exec/gdbstub.h"
 #include "exec/hwaddr.h"
 #include "exec/tb-flush.h"
 #include "exec/translation-block.h"
@@ -139,9 +140,11 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
         CPU_FOREACH(cpu) {
-            err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
-            if (err) {
-                break;
+            if (gdb_cpu_in_source_group(cs, cpu)) {
+                err = cpu_breakpoint_insert(cpu, addr, BP_GDB, NULL);
+                if (err) {
+                    break;
+                }
             }
         }
         return err;
@@ -149,10 +152,12 @@ static int tcg_insert_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
         CPU_FOREACH(cpu) {
-            err = cpu_watchpoint_insert(cpu, addr, len,
-                                        xlat_gdb_type(cpu, type), NULL);
-            if (err) {
-                break;
+            if (gdb_cpu_in_source_group(cs, cpu)) {
+                err = cpu_watchpoint_insert(cpu, addr, len,
+                                            xlat_gdb_type(cpu, type), NULL);
+                if (err) {
+                    break;
+                }
             }
         }
         return err;
@@ -170,9 +175,11 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_BREAKPOINT_SW:
     case GDB_BREAKPOINT_HW:
         CPU_FOREACH(cpu) {
-            err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
-            if (err) {
-                break;
+            if (gdb_cpu_in_source_group(cs, cpu)) {
+                err = cpu_breakpoint_remove(cpu, addr, BP_GDB);
+                if (err) {
+                    break;
+                }
             }
         }
         return err;
@@ -180,10 +187,12 @@ static int tcg_remove_breakpoint(CPUState *cs, int type, vaddr addr, vaddr len)
     case GDB_WATCHPOINT_READ:
     case GDB_WATCHPOINT_ACCESS:
         CPU_FOREACH(cpu) {
-            err = cpu_watchpoint_remove(cpu, addr, len,
-                                        xlat_gdb_type(cpu, type));
-            if (err) {
-                break;
+            if (gdb_cpu_in_source_group(cs, cpu)) {
+                err = cpu_watchpoint_remove(cpu, addr, len,
+                                            xlat_gdb_type(cpu, type));
+                if (err) {
+                    break;
+                }
             }
         }
         return err;
diff --git a/gdbstub/gdbstub.c b/gdbstub/gdbstub.c
index 0e2e10fbaa..652e0e768f 100644
--- a/gdbstub/gdbstub.c
+++ b/gdbstub/gdbstub.c
@@ -492,6 +492,12 @@ const GDBFeature *gdb_find_static_feature(const char *xmlname)
     g_assert_not_reached();
 }
 
+bool gdb_cpu_in_source_group(CPUState *cs, CPUState *cpu)
+{
+    return !gdbserver_state.multiprocess ||
+           (gdb_get_cpu_pid(cs) == gdb_get_cpu_pid(cpu));
+}
+
 GArray *gdb_get_register_list(CPUState *cpu)
 {
     GArray *results = g_array_new(true, true, sizeof(GDBRegDesc));
diff --git a/include/exec/gdbstub.h b/include/exec/gdbstub.h
index 0675b0b646..f92641793c 100644
--- a/include/exec/gdbstub.h
+++ b/include/exec/gdbstub.h
@@ -114,6 +114,18 @@ void gdb_feature_builder_end(const GDBFeatureBuilder *builder);
  */
 const GDBFeature *gdb_find_static_feature(const char *xmlname);
 
+/**
+ * Tests if the CPUs belong to the same group for the purposes of breakpoint
+ * insertion and deletion when running multiprocesses gdb. The test is only
+ * valid for multiprocess gdb and should not affect the insertion or deletion of
+ * breakpoints when we are not running in that mode.
+ * @cs: The CPU used as reference
+ * @cpu: The CPU to test
+ *
+ * Return: True if they belong to the same group or it is not a multiprocess gdb
+ */
+bool gdb_cpu_in_source_group(CPUState *cs, CPUState *cpu);
+
 /**
  * gdb_read_register() - Read a register associated with a CPU.
  * @cpu: The CPU associated with the register.
-- 
2.49.0.1015.ga840276032-goog