[PATCH 0/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[PATCH 0/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Wolfgang BETZ]

From: Wolfgang Betz <wolfgang.betz@st.com>

The Context ID Register, CONTEXTIDR, identifies the current:
- Process Identifier (PROCID) &
- Address Space Identifier (ASID).

The value of the whole of this register is called the Context ID and is 
used by:
- the ARM debug logic, for Linked and Unlinked Context ID matching
  (e.g. for breakpoint debug and watchpoint debug events).
- the trace logic, to identify the current process.

The CONTEXTIDR is a 32-bit read/write register whose format is:
- PROCID, bits [31:8]
  Process Identifier. This field should be programmed with a unique 
  value that identifies the current process and is used by the trace logic and 
  the debug logic to identify the process that is running currently.
- ASID, bits [7:0]
  Address Space Identifier. This field must be programmed with the 
  value of the current ASID and is used by many memory management functions.

This change-set aims at:
- implementing thread tracing support based on the armv6 & v7 CONTEXTIDR 
  register while leaving the Linux kernel ASID functionality (semantically)
  unchanged.
- focusing on compatibility with tracing tools such as Lauterbach's 
  TRACE32 tool.
- the avoidance of platform specific calls in generic code.
- simplicity, readability, and good performances.
- being general: i.e. the change-set applies to all armv7/v6 platforms &
  is in general compilable for all (other) platforms.

The patch has been jointly developed by 
Lauterbach GmbH (http://www.lauterbach.com) and 
STMicroelectronics Srl (http://www.st.com/). 
Main contributors are:
- Khaled Jmal <khaled.jmal@lauterbach.com>
- Rudi Dienstbeck <Rudolf.Dienstbeck@Lauterbach.com>
- Wolfgang Betz <wolfgang.betz@st.com>


Wolfgang Betz (1):
  Add Thread Support for the Context ID Register of ARM v6 & v7
    Architectures

 arch/arm/Kconfig.debug             |   14 +++++++++
 arch/arm/include/asm/mmu_context.h |   52 ++++++++++++++++++++++++++++++++---
 arch/arm/kernel/smp.c              |    2 +-
 arch/arm/mm/context.c              |   41 +++++++++++++++++++++++++++-
 arch/arm/mm/proc-v6.S              |    1 -
 arch/arm/mm/proc-v7.S              |    1 -
 6 files changed, 101 insertions(+), 10 deletions(-)

-- 
1.7.4.4

[PATCH 1/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Wolfgang BETZ]

From: Wolfgang Betz <wolfgang.betz@st.com>

The aim of this patch is to enable thread support in the context ID register
(CONTEXTIDR) as it comes with ARM architectures v6 & v7.

 On ARMv6 & v7, we have the following structure in the context ID:

   31                         7          0
   +-------------------------+-----------+
   |      process ID         |   ASID    |
   +-------------------------+-----------+
   |              context ID             |
   +-------------------------------------+

- The ASID is used to tag entries in the CPU caches and TLBs.
- The context ID is used by debuggers and trace logic, and
  should be unique within all running processes.

Currently the Linux kernel does correctly support the ASID field of the register,
but does not make use of the process ID in a way that would allow trace logic to
efficiently identify context switches.
In order to achieve this, this patch modifies 6 files of the kernel as described
hereafter:
- First a new configuration variable THREAD_CONTEXTID has been introduced in
  file "arch/arm/Kconfig.debug", which basically enables the patch (if not
  enabled, the kernel behaves as if it would not have been modified at all).
  This configuration variable depends obviously on the presence of a context
  ID register and automatically selects TRACING as the patch is partially based
  on tracepoints. Furthermore it enables both DEBUG_KERNEL and DEBUG_INFO as it
  is supposed that a backend tool which will analyze the generated trace will
  require access to debugging information like e.g. the debug symbols of the
  kernel and modules.
- The major part of the modifications of this patch are concentrated in file
  "arch/arm/include/asm/mmu_context.h", where a new function, i.e.
  "calc_context_id", has been introduced. The objective of this function is to
  calculate the contents of the context ID register (CONTEXTIDR), as described
  above, which should be moved into this register on the next context switch.
  Furthermore, there is a new convenience function, i.e. "set_context_id",
  which allows to set the CONTEXTIDR based on the outcome of a call to
  "calc_context_id". Finally, function "switch_mm" has been modified similarly
  by replacing the second argument in the call to "cpu_switch_mm" with the
  outcome of a call to "calc_context_id". The very same modification was
  necessary also in function "secondary_start_kernel" of file
  "arch/arm/kernel/smp.c".
- File "arch/arm/mm/context.c" has been modified for one thing to make use of the
  new convenience function "set_context_id", for another thing to register a new
  "sched_switch" tracepoint function to trace thread switches not already covered
  by "switch_mm".
- Finally, functions "cpu_v6_switch_mm" and "cpu_v7_switch_mm", in files
  "arch/arm/mm/proc-v6.S" and "arch/arm/mm/proc-v7.S" respectively, have been
  modified so that these expect now the content to be moved into CONTEXTIDR to be
  directly passed as second argument (i.e. within "r1").

Signed-off-by: Wolfgang Betz <wolfgang.betz@st.com>
---
 arch/arm/Kconfig.debug             |   14 +++++++++
 arch/arm/include/asm/mmu_context.h |   52 ++++++++++++++++++++++++++++++++---
 arch/arm/kernel/smp.c              |    2 +-
 arch/arm/mm/context.c              |   41 +++++++++++++++++++++++++++-
 arch/arm/mm/proc-v6.S              |    1 -
 arch/arm/mm/proc-v7.S              |    1 -
 6 files changed, 101 insertions(+), 10 deletions(-)

diff --git a/arch/arm/Kconfig.debug b/arch/arm/Kconfig.debug
index 81cbe40..dba51f7 100644
--- a/arch/arm/Kconfig.debug
+++ b/arch/arm/Kconfig.debug
@@ -129,4 +129,18 @@ config DEBUG_S3C_UART
 	  The uncompressor code port configuration is now handled
 	  by CONFIG_S3C_LOWLEVEL_UART_PORT.
 
+config THREAD_CONTEXTID
+	bool "Enable thread support for the Context ID Register"
+	depends on CPU_HAS_ASID
+	default n
+	select DEBUG_KERNEL
+	select DEBUG_INFO
+	select TRACING
+	help
+	  Say Y here if you want to enable thread support for the trace logic of
+	  tools such as Lauterbach's TRACE32 tool.
+	  
+	  This thread tracing support is based on the CONTEXTIDR register of
+	  architectures like the ARM v6 or v7.
+
 endmenu
diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 71605d9..3b26f71 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -68,8 +68,50 @@ static inline void check_context(struct mm_struct *mm)
 
 #define init_new_context(tsk,mm)	(__init_new_context(tsk,mm),0)
 
-#else
+#ifdef CONFIG_THREAD_CONTEXTID
+/*
+ * Calculate context ID for task and mm
+ */
+static inline struct mm_struct *calc_context_id(struct task_struct *tsk, struct mm_struct *mm)
+{
+	unsigned int ret;
+	
+	if (unlikely(tsk == NULL)) {
+		ret = (current->pid << ASID_BITS);
+	} else {
+		ret = (tsk->pid << ASID_BITS);
+	}
+	
+	if (unlikely(!ret)) {
+		ret = (0xFFFFFFFF << ASID_BITS);
+	}
+	
+	return (struct mm_struct *)((mm->context.id & ~ASID_MASK) | ret);
+}
+#else /* !CONFIG_THREAD_CONTEXTID */
+/*
+ * Calculate context ID for task and mm
+ */
+static inline struct mm_struct *calc_context_id(struct task_struct *tsk, struct mm_struct *mm)
+{
+	return (struct mm_struct *)(mm->context.id);
+}
+#endif /* !CONFIG_THREAD_CONTEXTID */
 
+/*
+ * Set context ID for task and mm
+ */
+static inline void set_context_id(struct task_struct *tsk, struct mm_struct *mm)
+{
+	unsigned int ctxid = (unsigned int)calc_context_id(tsk, mm);
+ 
+	/* set the new ContextID */
+	asm("mcr	p15, 0, %0, c13, c0, 1\n" : : "r" (ctxid));
+	isb();
+}
+ 
+#else // !CONFIG_CPU_HAS_ASID
+ 
 static inline void check_context(struct mm_struct *mm)
 {
 #ifdef CONFIG_MMU
@@ -79,9 +121,9 @@ static inline void check_context(struct mm_struct *mm)
 }
 
 #define init_new_context(tsk,mm)	0
-
-#endif
-
+#define calc_context_id(tsk,mm)		(mm)
+#endif // !CONFIG_CPU_HAS_ASID
+ 
 #define destroy_context(mm)		do { } while(0)
 
 /*
@@ -123,7 +165,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 		*crt_mm = next;
 #endif
 		check_context(next);
-		cpu_switch_mm(next->pgd, next);
+		cpu_switch_mm(next->pgd, calc_context_id(tsk, next));
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
 	}
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 344e52b..a518a39 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -288,7 +288,7 @@ asmlinkage void __cpuinit secondary_start_kernel(void)
 	atomic_inc(&mm->mm_count);
 	current->active_mm = mm;
 	cpumask_set_cpu(cpu, mm_cpumask(mm));
-	cpu_switch_mm(mm->pgd, mm);
+	cpu_switch_mm(mm->pgd, calc_context_id(current, mm));
 	enter_lazy_tlb(mm, current);
 	local_flush_tlb_all();
 
diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c
index b0ee9ba..a4976af 100644
--- a/arch/arm/mm/context.c
+++ b/arch/arm/mm/context.c
@@ -16,6 +16,10 @@
 #include <asm/mmu_context.h>
 #include <asm/tlbflush.h>
 
+#ifdef CONFIG_THREAD_CONTEXTID
+#include <trace/events/sched.h>
+#endif
+
 static DEFINE_SPINLOCK(cpu_asid_lock);
 unsigned int cpu_last_asid = ASID_FIRST_VERSION;
 #ifdef CONFIG_SMP
@@ -99,8 +103,7 @@ static void reset_context(void *info)
 	set_mm_context(mm, asid);
 
 	/* set the new ASID */
-	asm("mcr	p15, 0, %0, c13, c0, 1\n" : : "r" (mm->context.id));
-	isb();
+	set_context_id(current, mm);
 }
 
 #else
@@ -155,3 +158,37 @@ void __new_context(struct mm_struct *mm)
 	set_mm_context(mm, asid);
 	spin_unlock(&cpu_asid_lock);
 }
+
+#ifdef CONFIG_THREAD_CONTEXTID
+/*
+ * Add support for threads in CONTEXTIDR by registering a 
+ * 'sched_switch' tracepoint event function
+ */
+static void thrctx_sched_switch(void *ignore, struct task_struct *prev, struct task_struct *next)
+{
+	struct mm_struct *mm, *oldmm;
+
+	mm = next->mm;
+	oldmm = prev->active_mm;
+
+	if (!mm) {
+		set_context_id(next, oldmm);
+	} else {
+		if (oldmm == mm)
+			set_context_id(next, mm);
+	}
+}
+
+__init static int init_thread_contextid(void)
+{
+	int ret;
+
+	ret = register_trace_sched_switch(thrctx_sched_switch, NULL);
+	if (ret)
+		pr_info("ftrace_graph: Couldn't activate tracepoint"
+			" probe to kernel_sched_switch\n");
+
+	return ret;
+}
+device_initcall(init_thread_contextid);
+#endif /* CONFIG_THREAD_CONTEXTID */
diff --git a/arch/arm/mm/proc-v6.S b/arch/arm/mm/proc-v6.S
index 1d2b845..57f3574 100644
--- a/arch/arm/mm/proc-v6.S
+++ b/arch/arm/mm/proc-v6.S
@@ -93,7 +93,6 @@ ENTRY(cpu_v6_dcache_clean_area)
 ENTRY(cpu_v6_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
-	ldr	r1, [r1, #MM_CONTEXT_ID]	@ get mm->context.id
 	ALT_SMP(orr	r0, r0, #TTB_FLAGS_SMP)
 	ALT_UP(orr	r0, r0, #TTB_FLAGS_UP)
 	mcr	p15, 0, r2, c7, c5, 6		@ flush BTAC/BTB
diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index 3c38678..a51097a 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -102,7 +102,6 @@ ENDPROC(cpu_v7_dcache_clean_area)
 ENTRY(cpu_v7_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
-	ldr	r1, [r1, #MM_CONTEXT_ID]	@ get mm->context.id
 	ALT_SMP(orr	r0, r0, #TTB_FLAGS_SMP)
 	ALT_UP(orr	r0, r0, #TTB_FLAGS_UP)
 #ifdef CONFIG_ARM_ERRATA_430973
-- 
1.7.4.4

[PATCH 1/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Russell King - ARM Linux]

On Mon, Jun 27, 2011 at 01:12:27PM +0200, Wolfgang BETZ wrote:
> The aim of this patch is to enable thread support in the context ID register
> (CONTEXTIDR) as it comes with ARM architectures v6 & v7.
> 
>  On ARMv6 & v7, we have the following structure in the context ID:
> 
>    31                         7          0
>    +-------------------------+-----------+
>    |      process ID         |   ASID    |
>    +-------------------------+-----------+
>    |              context ID             |
>    +-------------------------------------+
> 
> - The ASID is used to tag entries in the CPU caches and TLBs.
> - The context ID is used by debuggers and trace logic, and
>   should be unique within all running processes.

When I added ARMv6 and v7 support to the kernel, I purposely ignored that
detail because it's silly.

The issue here is that the debuggers and trace logic use both the
process ID and the ASID together, and the ASID does not have a stable
value for any particular process in the system.  When we have used
the last ASID (255), the next request for a new ASID causes the ASID
generation to increment, which in turn causes all processes in the
system to have their ASIDs reassigned.

At that point, if you have debugging/trace hardware setup to look for
a process with a particular process ID+ASID, you're no longer going to
get matches for your original process.

This makes the whole concept of debuggers/tracing using the context ID
rather pointless especially in a multiprocessor system.

How are you working around this issue?

[PATCH 1/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Will Deacon]

Hi Russell,

On Mon, Jun 27, 2011 at 12:37:08PM +0100, Russell King - ARM Linux wrote:
> On Mon, Jun 27, 2011 at 01:12:27PM +0200, Wolfgang BETZ wrote:
> > The aim of this patch is to enable thread support in the context ID register
> > (CONTEXTIDR) as it comes with ARM architectures v6 & v7.
> > 
> >  On ARMv6 & v7, we have the following structure in the context ID:
> > 
> >    31                         7          0
> >    +-------------------------+-----------+
> >    |      process ID         |   ASID    |
> >    +-------------------------+-----------+
> >    |              context ID             |
> >    +-------------------------------------+
> > 
> > - The ASID is used to tag entries in the CPU caches and TLBs.
> > - The context ID is used by debuggers and trace logic, and
> >   should be unique within all running processes.
> 
> When I added ARMv6 and v7 support to the kernel, I purposely ignored that
> detail because it's silly.

[...]

> How are you working around this issue?

I don't know how you could get around this with current implementations, but
for Cortex-A15 w/ LPAE the ASID lives in the TTBR rather than the context
ID register, so using the PID is straightforward in that case.

Will

[PATCH 1/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Wolfgang BETZ]

Ciao Will,

thanks a lot for your statement regarding the ASID overflow issue.
In fact, we have never even tried to resolve this issue, not only 
because most likely there is no perfect solution to it in current 
implementations (i.e. v6 & v7 architectures), but also because it was 
out of our scope.
In STMicroelectronics we are simply customers of Lauterbach's TRACE32 
tool and are very interested in getting an as precise as possible 
insight in what goes on in our systems when running Linux based 
environments, with a particular focus on SMP. Together with Lauterbach 
we are currently working on improving their tool with respect to SMP in 
general and the tracing of context switches in particular. One result of 
this collaboration is the patch I have submitted to the ARM kernel 
mailing list this Monday.
Again, we do not aim in improving the handling of ASIDs, but are just 
caring about getting also the current process ID out of the trace logic. 
As the way to achieve this is not platform-specific, we thought it might 
be a good idea to implement it as an architectural feature and provide 
it through a kernel configuration variable (THREAD_CONTEXTID) for any 
platform supporting the CONTEXTIDR.
Furthermore, we thought it would be a good thing to mainline this 
modification and provide it to the open-source community. I do not know 
if you had the time to take a closer look to our patch, but for sure 
your feedback would be more than welcome.

Best regards,
Wolfgang


On 06/28/2011 11:00 AM, Will Deacon wrote:
> Hi Russell,
>
> On Mon, Jun 27, 2011 at 12:37:08PM +0100, Russell King - ARM Linux wrote:
>> On Mon, Jun 27, 2011 at 01:12:27PM +0200, Wolfgang BETZ wrote:
>>> The aim of this patch is to enable thread support in the context ID register
>>> (CONTEXTIDR) as it comes with ARM architectures v6&  v7.
>>>
>>>   On ARMv6&  v7, we have the following structure in the context ID:
>>>
>>>     31                         7          0
>>>     +-------------------------+-----------+
>>>     |      process ID         |   ASID    |
>>>     +-------------------------+-----------+
>>>     |              context ID             |
>>>     +-------------------------------------+
>>>
>>> - The ASID is used to tag entries in the CPU caches and TLBs.
>>> - The context ID is used by debuggers and trace logic, and
>>>    should be unique within all running processes.
>> When I added ARMv6 and v7 support to the kernel, I purposely ignored that
>> detail because it's silly.
> [...]
>
>> How are you working around this issue?
> I don't know how you could get around this with current implementations, but
> for Cortex-A15 w/ LPAE the ASID lives in the TTBR rather than the context
> ID register, so using the PID is straightforward in that case.
>
> Will
>

[PATCH 1/1] Add Thread Support for the Context ID Register of ARM v6 & v7 Architectures

[Russell King - ARM Linux]

On Tue, Jun 28, 2011 at 08:57:28AM +0200, Wolfgang BETZ wrote:
> First of all, the case you are mentioning is very rarely met in embedded
> systems, where we are dealing with less processes - not hundreds.
> Furthermore, we are tracing seconds to minutes. So usually this is not
> an issue for Lauterbach and above all their customers. They already
> have several customers using this patch - without running into this
> trouble.

It is not about how many processes are running in total, it is about
how many processes you are running _and_ how many you've started up.

Each time a thread is created via fork(), or it executes a new program
via execve(), a new ASID number will be assigned to the new context.
So, each time a process decides to execute something as a sub-process,
you will eat through two ASID numbers.

This means that if your system boot has run around 120 programs, you
are very close to an ASID reallocation event, and if you are tracing
or debugging at that time using this facility, all your context IDs
immediately become invalid.

If your system does not involve executing programs after boot, then
you are safe, but I suspect that is only true of a very small minority
of embedded systems.

> In any case - *if* we get into this, it's very simple in TRACE32 to
> ignore the ASID while extracting the ContextID from the trace - the
> PID should be unique anyway. Therefore we think it is more a problem
> of the back-end tool which elaborates the trace.

My concern is that you may be tempted to use CONTEXTIDR to control
debugging activity, users will set a breakpoint and (to them) for some
unknown reason their breakpoint isn't hit because the ASIDs were
reallocated.

To me it just feels extremely fragile, leads to surprises, and I _really_
don't like that.  I've been on the customer end of fragile embedded tools,
and I know full well that its these kinds of things that piss customers
off and waste their time.

I've spent more time debugging CPU ICE systems than debugging my programs
which are supposed to be running on those CPUs - and had to because the
real CPU was an OTP device.

So, when presented with a patch which is inherently fragile, I'm going
to say no to it given my personal frustrating experiences of fragile
embedded tools.

I hope that by saying no to this kind of thing I can save some other poor
embedded engineer many hours of head-scratching wondering why the
expensive tool they bought/hired doesn't actually work.