* Relax tracing not working for me @ 2025-01-07 22:51 Rosenow, James 2025-01-08 5:26 ` Chen, Hongzhan 2025-01-08 9:36 ` Philippe Gerum 0 siblings, 2 replies; 6+ messages in thread From: Rosenow, James @ 2025-01-07 22:51 UTC (permalink / raw) To: xenomai@lists.linux.dev I am using Xenomai v3.2.1 on 5.15.57-dovetail and I have selected the following kernel configuration option. .config - Linux/x86 5.15.57 Kernel Configuration Xenomai/cobalt > Debug support [*] Trace relax requests When I run my application, I am getting MSW's on one of my real-time tasks, i.e. seqTask-00 (below). Every 10.0s: cat /proc/xenomai/sched/stat test: Tue Jan 7 16:32:14 2025 CPU PID MSW CSW XSC PF STAT %CPU NAME 0 0 0 3526511 0 0 00018000 99.8 [ROOT/0] 1 0 0 22166547 0 0 00018000 98.7 [ROOT/1] 2 0 0 6 0 0 00018000 100.0 [ROOT/2] 3 0 0 0 0 0 00018000 100.0 [ROOT/3] 4 0 0 12 0 0 00018000 100.0 [ROOT/4] 5 0 0 0 0 0 00018000 100.0 [ROOT/5] 0 311 0 2 0 0 00000042 0.0 [rtnet-stack] 0 312 0 2 0 0 00020042 0.0 [rtnet-rtpc] 0 34201 8 8 54 0 000480c0 0.0 coops_target_main 0 34203 714420 714421 1429756 0 000480c0 0.2 userCycle 1 34204 2 2 571184 0 000480c0 0.0 clockInterrupt 1 34205 1 571183 1356559 0 00048042 0.4 dispatch 1 34206 571182 1142364 713982 0 00048042 0.5 seqTask-00 1 34207 1 285594 356993 0 00048042 0.3 seqTask-01 1 34208 1 142798 180963 0 00048042 0.1 seqTask-02 1 34209 1 2 3 0 00048042 0.0 seqTask-03 1 34210 1 2 3 0 00048042 0.0 seqTask-04 This is an iterative task that is executing at 2000Hz, I get 1 additional MSW for each run iteration. As you can see there are many MSW's. Shouldn't the trace buffer be full of these relaxations? If I cat the buffer, I see nothing. test@test:~$ cat /proc/xenomai/debug/relax test@test:~$ What am I doing wrong? Thanks and regards, Jim Rosenow MTS Systems Corporation ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Relax tracing not working for me 2025-01-07 22:51 Relax tracing not working for me Rosenow, James @ 2025-01-08 5:26 ` Chen, Hongzhan 2025-01-08 5:53 ` Chen, Hongzhan 2025-01-08 9:36 ` Philippe Gerum 1 sibling, 1 reply; 6+ messages in thread From: Chen, Hongzhan @ 2025-01-08 5:26 UTC (permalink / raw) To: Rosenow, James, xenomai@lists.linux.dev >-----Original Message----- >From: Rosenow, James <jim.rosenow@mts.com> >Sent: Wednesday, January 8, 2025 6:51 AM >To: xenomai@lists.linux.dev >Subject: Relax tracing not working for me > >I am using Xenomai v3.2.1 on 5.15.57-dovetail and I have selected the >following kernel configuration option. > >.config - Linux/x86 5.15.57 Kernel Configuration > Xenomai/cobalt > Debug support >[*] Trace relax requests > >When I run my application, I am getting MSW's on one of my real-time tasks, >i.e. seqTask-00 (below). > >Every 10.0s: cat /proc/xenomai/sched/stat test: Tue Jan 7 >16:32:14 2025 >CPU PID MSW CSW XSC PF STAT %CPU NAME > 0 0 0 3526511 0 0 00018000 99.8 [ROOT/0] > 1 0 0 22166547 0 0 00018000 98.7 [ROOT/1] > 2 0 0 6 0 0 00018000 100.0 [ROOT/2] > 3 0 0 0 0 0 00018000 100.0 [ROOT/3] > 4 0 0 12 0 0 00018000 100.0 [ROOT/4] > 5 0 0 0 0 0 00018000 100.0 [ROOT/5] > 0 311 0 2 0 0 00000042 0.0 [rtnet-stack] > 0 312 0 2 0 0 00020042 0.0 [rtnet-rtpc] > 0 34201 8 8 54 0 000480c0 0.0 coops_target_main > 0 34203 714420 714421 1429756 0 000480c0 0.2 userCycle > 1 34204 2 2 571184 0 000480c0 0.0 clockInterrupt > 1 34205 1 571183 1356559 0 00048042 0.4 dispatch > 1 34206 571182 1142364 713982 0 00048042 0.5 seqTask-00 > 1 34207 1 285594 356993 0 00048042 0.3 seqTask-01 > 1 34208 1 142798 180963 0 00048042 0.1 seqTask-02 > 1 34209 1 2 3 0 00048042 0.0 seqTask-03 > 1 34210 1 2 3 0 00048042 0.0 seqTask-04 > >This is an iterative task that is executing at 2000Hz, I get 1 additional MSW for >each run iteration. As you can see there are many MSW's. Shouldn't the trace >buffer be full of these relaxations? If I cat the buffer, I see nothing. > >test@test:~$ cat /proc/xenomai/debug/relax >test@test:~$ > >What am I doing wrong? According to the code [1] of xnthread_relax, system only record relax that notify is not zero. After search xnthread_relax that pass notify=1, we can find that only the xnthread_relax called in kernel/cobalt/posix/syscall.c will try to set notify=1. That means that /proc/xenomai/debug/relax only record relax which caused by system call that try to switch primary domain to secondary domain. All the relaxes that you observed via MSW may not be recorded. Regards Hongzhan Chen [1]: if (xnthread_test_state(thread, XNUSER) && notify) { if (xnthread_test_state(thread, XNWARN)) { /* Help debugging spurious relaxes. */ xndebug_notify_relax(thread, reason); memset(&si, 0, sizeof(si)); si.si_signo = SIGDEBUG; si.si_code = SI_QUEUE; si.si_int = reason | sigdebug_marker; send_sig_info(SIGDEBUG, &si, p); } xnsynch_detect_boosted_relax(thread); } > >Thanks and regards, >Jim Rosenow >MTS Systems Corporation ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Relax tracing not working for me 2025-01-08 5:26 ` Chen, Hongzhan @ 2025-01-08 5:53 ` Chen, Hongzhan 0 siblings, 0 replies; 6+ messages in thread From: Chen, Hongzhan @ 2025-01-08 5:53 UTC (permalink / raw) To: Chen, Hongzhan, Rosenow, James, xenomai@lists.linux.dev I tried to run netcat -l -p 67676 -c < /proc/xenomai/debug/relax as introduced in https://v3.xenomai.org/tips/spurious-relaxes/ to capture relax, but failed with following error. netcat: invalid option -- 'c' usage: nc [-46CDdFhklNnrStUuvZz] [-I length] [-i interval] [-M ttl] [-m minttl] [-O length] [-P proxy_username] [-p source_port] [-q seconds] [-s sourceaddr] [-T keyword] [-V rtable] [-W recvlimit] [-w timeout] [-X proxy_protocol] [-x proxy_address[:port]] [destination] [port] Regards Hongzhan Chen >-----Original Message----- >From: Chen, Hongzhan <hongzhan.chen@intel.com> >Sent: Wednesday, January 8, 2025 1:27 PM >To: Rosenow, James <jim.rosenow@mts.com>; xenomai@lists.linux.dev >Subject: RE: Relax tracing not working for me > > > >>-----Original Message----- >>From: Rosenow, James <jim.rosenow@mts.com> >>Sent: Wednesday, January 8, 2025 6:51 AM >>To: xenomai@lists.linux.dev >>Subject: Relax tracing not working for me >> >>I am using Xenomai v3.2.1 on 5.15.57-dovetail and I have selected the >>following kernel configuration option. >> >>.config - Linux/x86 5.15.57 Kernel Configuration >> Xenomai/cobalt > Debug support >>[*] Trace relax requests >> >>When I run my application, I am getting MSW's on one of my real-time tasks, >>i.e. seqTask-00 (below). >> >>Every 10.0s: cat /proc/xenomai/sched/stat test: Tue Jan 7 >>16:32:14 2025 >>CPU PID MSW CSW XSC PF STAT %CPU NAME >> 0 0 0 3526511 0 0 00018000 99.8 [ROOT/0] >> 1 0 0 22166547 0 0 00018000 98.7 [ROOT/1] >> 2 0 0 6 0 0 00018000 100.0 [ROOT/2] >> 3 0 0 0 0 0 00018000 100.0 [ROOT/3] >> 4 0 0 12 0 0 00018000 100.0 [ROOT/4] >> 5 0 0 0 0 0 00018000 100.0 [ROOT/5] >> 0 311 0 2 0 0 00000042 0.0 [rtnet-stack] >> 0 312 0 2 0 0 00020042 0.0 [rtnet-rtpc] >> 0 34201 8 8 54 0 000480c0 0.0 coops_target_main >> 0 34203 714420 714421 1429756 0 000480c0 0.2 userCycle >> 1 34204 2 2 571184 0 000480c0 0.0 clockInterrupt >> 1 34205 1 571183 1356559 0 00048042 0.4 dispatch >> 1 34206 571182 1142364 713982 0 00048042 0.5 seqTask-00 >> 1 34207 1 285594 356993 0 00048042 0.3 seqTask-01 >> 1 34208 1 142798 180963 0 00048042 0.1 seqTask-02 >> 1 34209 1 2 3 0 00048042 0.0 seqTask-03 >> 1 34210 1 2 3 0 00048042 0.0 seqTask-04 >> >>This is an iterative task that is executing at 2000Hz, I get 1 additional MSW for >>each run iteration. As you can see there are many MSW's. Shouldn't the >trace >>buffer be full of these relaxations? If I cat the buffer, I see nothing. >> >>test@test:~$ cat /proc/xenomai/debug/relax >>test@test:~$ >> >>What am I doing wrong? > >According to the code [1] of xnthread_relax, system only record relax that >notify is not zero. >After search xnthread_relax that pass notify=1, we can find that only the >xnthread_relax called in >kernel/cobalt/posix/syscall.c will try to set notify=1. That means that >/proc/xenomai/debug/relax only record relax which caused by system call that >try to switch primary domain to secondary domain. > >All the relaxes that you observed via MSW may not be recorded. > >Regards > >Hongzhan Chen > >[1]: > if (xnthread_test_state(thread, XNUSER) && notify) { > if (xnthread_test_state(thread, XNWARN)) { > /* Help debugging spurious relaxes. */ > xndebug_notify_relax(thread, reason); > memset(&si, 0, sizeof(si)); > si.si_signo = SIGDEBUG; > si.si_code = SI_QUEUE; > si.si_int = reason | sigdebug_marker; > send_sig_info(SIGDEBUG, &si, p); > } > xnsynch_detect_boosted_relax(thread); > } >> >>Thanks and regards, >>Jim Rosenow >>MTS Systems Corporation ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Relax tracing not working for me 2025-01-07 22:51 Relax tracing not working for me Rosenow, James 2025-01-08 5:26 ` Chen, Hongzhan @ 2025-01-08 9:36 ` Philippe Gerum 2025-01-09 19:48 ` Relax tracing not working for me (solved) Rosenow, James 1 sibling, 1 reply; 6+ messages in thread From: Philippe Gerum @ 2025-01-08 9:36 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: > I am using Xenomai v3.2.1 on 5.15.57-dovetail and I have selected the following kernel configuration option. > > .config - Linux/x86 5.15.57 Kernel Configuration > Xenomai/cobalt > Debug support > [*] Trace relax requests > > When I run my application, I am getting MSW's on one of my real-time tasks, i.e. seqTask-00 (below). > > Every 10.0s: cat /proc/xenomai/sched/stat test: Tue Jan 7 16:32:14 2025 > CPU PID MSW CSW XSC PF STAT %CPU NAME > 0 0 0 3526511 0 0 00018000 99.8 [ROOT/0] > 1 0 0 22166547 0 0 00018000 98.7 [ROOT/1] > 2 0 0 6 0 0 00018000 100.0 [ROOT/2] > 3 0 0 0 0 0 00018000 100.0 [ROOT/3] > 4 0 0 12 0 0 00018000 100.0 [ROOT/4] > 5 0 0 0 0 0 00018000 100.0 [ROOT/5] > 0 311 0 2 0 0 00000042 0.0 [rtnet-stack] > 0 312 0 2 0 0 00020042 0.0 [rtnet-rtpc] > 0 34201 8 8 54 0 000480c0 0.0 coops_target_main > 0 34203 714420 714421 1429756 0 000480c0 0.2 userCycle > 1 34204 2 2 571184 0 000480c0 0.0 clockInterrupt > 1 34205 1 571183 1356559 0 00048042 0.4 dispatch > 1 34206 571182 1142364 713982 0 00048042 0.5 seqTask-00 > 1 34207 1 285594 356993 0 00048042 0.3 seqTask-01 > 1 34208 1 142798 180963 0 00048042 0.1 seqTask-02 > 1 34209 1 2 3 0 00048042 0.0 seqTask-03 > 1 34210 1 2 3 0 00048042 0.0 seqTask-04 > > This is an iterative task that is executing at 2000Hz, I get 1 additional MSW for each run iteration. As you can see there are many MSW's. Shouldn't the trace buffer be full of these relaxations? If I cat the buffer, I see nothing. > > test@test:~$ cat /proc/xenomai/debug/relax > test@test:~$ > > What am I doing wrong? > In short, you may want to make sure that PTHREAD_WARNSW is set for 'seqTask-00'. The STAT field for that task reveals that it's not (0x4000 is clear there). Why? Because the logic of MSW tracing is as follows when CONFIG_XENO_OPT_DEBUG_TRACE_RELAX is set: 1. On MSW detected by the core for the current thread _and only if_ PTHREAD_WARNSW is set for such thread (which is running a syscall), the latter is immediately sent SIGWINCH(aka SIGSHADOW), with info=SIGSHADOW_ACTION_BACKTRACE (xnthread_relax -> xndebug_notify_relax). 2. The pre-defined SIGSHADOW handler in libcobalt.so (see lib/cobalt/sigshadow.c) runs, taking a snapshot of the current stack backtrace, then sends it to the core via Cobalt's 'backtrace' system call. Since we are not using alternate stacks, the code which caused the MSW must appear somewhere in the lower frames below the current signal frame. This location is figured out by the slackspot utility when analyzing the captured frames. 3. The core receives the backtrace data, then logs it into the appropriate trace buffer which is behind /proc/xenomai/debug/relax (backtrace -> xndebug_trace_relax, cobalt/posix/syscall.c). HTH, -- Philippe. ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Relax tracing not working for me (solved) 2025-01-08 9:36 ` Philippe Gerum @ 2025-01-09 19:48 ` Rosenow, James 2025-01-11 18:07 ` Philippe Gerum 0 siblings, 1 reply; 6+ messages in thread From: Rosenow, James @ 2025-01-09 19:48 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai@lists.linux.dev Thank you, Philippe. This helped; I have located the call source of the MSW and now I am in need some guidance on how to go about fixing it. The offending calls look like this: Userspace call: int dma_move( struct ioctl_dma_req *dr_p) { return( ioctl( fd, IOCTL_DMA_MOVE, dr_p)); } Kernel driver call: int ioctl_dma( struct dma_device *dev, unsigned int cmd, unsigned long arg) { int retval; retval = 0; switch ( cmd) { case IOCTL_DMA_MOVE: { struct ioctl_dma_req dma_req; if( copy_from_user(&dma_req, (void *)arg, sizeof(dma_req))) { return( -EFAULT); } retval = dma_move( dev, &dma_req); if( retval < 0) { return( retval); } if( copy_to_user( (void *)arg, &dma_req, sizeof( dma_req))) { return -EFAULT; } break; } So my questions are these: 1) Is the MSW occurring due to the ioctl() call from userspace? 2) Or is the MSW occurring in the kernel module? And finally, 3) In this scenario, which must be somewhat common for real-time systems, i.e., a device driver is needed to facilitate some sort of real-time hardware operation, how does one 'port' this properly into Xenomai such that it does incur MSW? Thanks and regards, Jim Rosenow MTS Systems Corporation > -----Original Message----- > From: Philippe Gerum <rpm@xenomai.org> > Sent: Wednesday, January 8, 2025 3:36 AM > To: Rosenow, James <jim.rosenow@mts.com> > Cc: xenomai@lists.linux.dev > Subject: Re: Relax tracing not working for me > > CAUTION: External Email > > > "Rosenow, James" <jim.rosenow@mts.com> writes: > > > I am using Xenomai v3.2.1 on 5.15.57-dovetail and I have selected the following > kernel configuration option. > > > > .config - Linux/x86 5.15.57 Kernel Configuration Xenomai/cobalt > > > Debug support > > [*] Trace relax requests > > > > When I run my application, I am getting MSW's on one of my real-time tasks, i.e. > seqTask-00 (below). > > > > Every 10.0s: cat /proc/xenomai/sched/stat test: Tue Jan 7 > 16:32:14 2025 > > CPU PID MSW CSW XSC PF STAT %CPU NAME > > 0 0 0 3526511 0 0 00018000 99.8 [ROOT/0] > > 1 0 0 22166547 0 0 00018000 98.7 [ROOT/1] > > 2 0 0 6 0 0 00018000 100.0 [ROOT/2] > > 3 0 0 0 0 0 00018000 100.0 [ROOT/3] > > 4 0 0 12 0 0 00018000 100.0 [ROOT/4] > > 5 0 0 0 0 0 00018000 100.0 [ROOT/5] > > 0 311 0 2 0 0 00000042 0.0 [rtnet-stack] > > 0 312 0 2 0 0 00020042 0.0 [rtnet-rtpc] > > 0 34201 8 8 54 0 000480c0 0.0 coops_target_main > > 0 34203 714420 714421 1429756 0 000480c0 0.2 userCycle > > 1 34204 2 2 571184 0 000480c0 0.0 clockInterrupt > > 1 34205 1 571183 1356559 0 00048042 0.4 dispatch > > 1 34206 571182 1142364 713982 0 00048042 0.5 seqTask-00 > > 1 34207 1 285594 356993 0 00048042 0.3 seqTask-01 > > 1 34208 1 142798 180963 0 00048042 0.1 seqTask-02 > > 1 34209 1 2 3 0 00048042 0.0 seqTask-03 > > 1 34210 1 2 3 0 00048042 0.0 seqTask-04 > > > > This is an iterative task that is executing at 2000Hz, I get 1 additional MSW for > each run iteration. As you can see there are many MSW's. Shouldn't the trace > buffer be full of these relaxations? If I cat the buffer, I see nothing. > > > > test@test:~$ cat /proc/xenomai/debug/relax test@test:~$ > > > > What am I doing wrong? > > > > In short, you may want to make sure that PTHREAD_WARNSW is set for 'seqTask- > 00'. The STAT field for that task reveals that it's not (0x4000 is clear there). > > Why? Because the logic of MSW tracing is as follows when > CONFIG_XENO_OPT_DEBUG_TRACE_RELAX is set: > > 1. On MSW detected by the core for the current thread _and only if_ > PTHREAD_WARNSW is set for such thread (which is running a syscall), the latter is > immediately sent SIGWINCH(aka SIGSHADOW), with > info=SIGSHADOW_ACTION_BACKTRACE (xnthread_relax -> > xndebug_notify_relax). > > 2. The pre-defined SIGSHADOW handler in libcobalt.so (see > lib/cobalt/sigshadow.c) runs, taking a snapshot of the current stack backtrace, > then sends it to the core via Cobalt's 'backtrace' system call. Since we are not > using alternate stacks, the code which caused the MSW must appear somewhere > in the lower frames below the current signal frame. This location is figured out by > the slackspot utility when analyzing the captured frames. > > 3. The core receives the backtrace data, then logs it into the appropriate trace > buffer which is behind /proc/xenomai/debug/relax (backtrace -> > xndebug_trace_relax, cobalt/posix/syscall.c). > > HTH, > > -- > Philippe. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Relax tracing not working for me (solved) 2025-01-09 19:48 ` Relax tracing not working for me (solved) Rosenow, James @ 2025-01-11 18:07 ` Philippe Gerum 0 siblings, 0 replies; 6+ messages in thread From: Philippe Gerum @ 2025-01-11 18:07 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: > Thank you, Philippe. > > This helped; I have located the call source of the MSW and now I am in need some guidance on how to go about fixing it. > > The offending calls look like this: > > Userspace call: > > int dma_move( struct ioctl_dma_req *dr_p) > { > return( ioctl( fd, IOCTL_DMA_MOVE, dr_p)); > } > > Kernel driver call: > > int > ioctl_dma( struct dma_device *dev, > unsigned int cmd, > unsigned long arg) > { > int retval; > > retval = 0; > switch ( cmd) > { > case IOCTL_DMA_MOVE: > { > struct ioctl_dma_req dma_req; > > if( copy_from_user(&dma_req, (void *)arg, sizeof(dma_req))) > { > return( -EFAULT); > } > retval = dma_move( dev, &dma_req); > if( retval < 0) > { > return( retval); > } > if( copy_to_user( (void *)arg, &dma_req, sizeof( dma_req))) > { > return -EFAULT; > } > break; > } > > So my questions are these: > 1) Is the MSW occurring due to the ioctl() call from userspace? Yes. > 2) Or is the MSW occurring in the kernel module? > There can be no _implicit_ MSW from kernel space, because that one may only happen on crossing the user->kernel boundary when handling a system call, depending on the syscall type. This means that calling in-band code from oob context past this crossing is always unchecked, unsafe, and the fastest path to train wreck. > And finally, > 3) In this scenario, which must be somewhat common for real-time systems, i.e., a device driver is needed to facilitate some sort of real-time hardware operation, how does one 'port' this properly into Xenomai such that it does incur MSW? > The answer mostly depends on the device, so let's focus on DMA ops: a) the DMA hardware is known from - and managed by - the regular 'dmaengine' framework available from the kernel. IOW, this peripheral may be providing data transfer services to some (inband) driver already. In this case, one would have to provide oob support to the corresponding DMA back-end driver which deals with the nitty-gritty details of the DMA hardware. This kind of support code is currently provided by Dovetail [1] for the bcm2835-dma and imx-sdma [2] back-ends. So, in order to share a DMA device between inband and oob activities seamlessly for (kernel-based) users (like dma_move() in your driver), one has to implement the oob extension briefly described in the doc I mentioned. There is no other sane option for addressing this case. Actually, there is no other working option, period. Also, I'm assuming a Dovetail-based setup, which provides the required infrastructure for implementing such extension. Legacy I-pipe is not an option in this case either. b) the DMA engine is hosted on some custom hardware of your own, which is not available to common kernel users. IOW, only your driver would know about it. This is the simplest case: you would implement the DMA transfer ops using oob-compatible constructs (e.g. hard spinlocks for serialization), according to your DMA interface specs, easy peasy. That would even be compatible with concurrent inband use with proper serialization. In that case, some driver of yours would provide the DMA management code for that particular DMA engine, and dma_move() would call that code (or implement such code, for that matter). c) the two options above assume that you have real-time requirements for DMA ops, since you mentioned 'real-time hardware operations'. If the use case is actually more relaxed, i.e. you need to kick the DMA transfer from some oob thread, but you could live with some unbounded delay until the transfer completes, then there is the option for any oob code to offload the DMA transfer to a routine which would run inband "asap". That routine would use the regular dmaengine calls to manage the transfer, no need for oob extension of the DMA back-end involved then. [1] https://v4.xenomai.org/core/drivers/dma/ [2] https://source.denx.de/Xenomai/linux-dovetail/-/blob/v6.1.y-cip-dovetail/drivers/dma/imx-sdma.c?ref_type=heads (see CONFIG_IMX_SDMA_OOB) -- Philippe. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-01-11 18:08 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-01-07 22:51 Relax tracing not working for me Rosenow, James 2025-01-08 5:26 ` Chen, Hongzhan 2025-01-08 5:53 ` Chen, Hongzhan 2025-01-08 9:36 ` Philippe Gerum 2025-01-09 19:48 ` Relax tracing not working for me (solved) Rosenow, James 2025-01-11 18:07 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.