[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[Anton Vorontsov]

Hi all,

These patches introduce KGDB FIQ debugger support. The idea (and some
code, of course) comes from Google's FIQ debugger[1]. There are some
differences (mostly implementation details, feature-wise they're almost
equivalent, or can be made equivalent, if desired).

The FIQ debugger is a facility that can be used to debug situations
when the kernel stuck in uninterruptable sections, e.g. the kernel
infinitely loops or deadlocked in an interrupt or with interrupts
disabled. On some development boards there is even a special NMI
button, which is very useful for debugging weird kernel hangs.

And FIQ is basically an NMI, it has a higher priority than IRQs, and
upon IRQ exception FIQs are not disabled. It is still possible to
disable FIQs (as well as some "NMIs" on other architectures), but via
special means.

So, here FIQs and NMIs are synonyms, but in the code I use NMI term
for arch-independent code, and FIQs for ARM code.

A few years ago KDB wasn't yet ready for production, or even not
well-known, so originally Google implemented its own FIQ debugger
that included its own shell, ring-buffer, commands, dumping,
backtracing logic and whatnot. This is very much like PowerPC's xmon
(arch/powerpc/xmon), except that xmon was there for a decade, so it
even predates KDB.

Anyway, nowadays KGDB/KDB is the cross-platform debugger, and the
only feature that was missing is NMI handling. This is now fixed for
ARM.

There a few differences comparing to the original (Google's) FIQ
debugger:

- Doing stuff in FIQ context is dangerous, as there we are not allowed
  to cause aborts or faults. In the original FIQ debugger there was a
  "signal" software-induced interrupt, upon exit from FIQ it would fire,
  and we would continue to execute "dangerous" commands from there.

  In KGDB/KDB we don't use signal interrupts. We can do easier:
  set up a breakpoint, continue, and you'll trap into KGDB again
  in a safe context.

  It works for most cases, but I can imagine cases when you can't
  set up a breakpoint. For these cases we'd better introduce a
  KDB command "exit_nmi", that will rise the SW IRQ, after which
  we're allowed to do anything.

- KGDB/KDB FIQ debugger shell is synchronous. In Google's version
  you could have a dedicated shell always running in the FIQ context,
  so when you type something on a serial line, you won't actually cause
  any debugging actions, FIQ would save the characters in its own
  buffer and continue execution normally. But when you hit return key
  after the command, then the command is executed.

  In KGDB/KDB FIQ debugger it is different. When you start any activity
  on the FIQ-enabled serial console, you'll enter KGDB and kernel will
  stop until you instruct it to continue.

  This might look as a drastic change, but it is not. There is actually
  no difference whether you have sync or async shell, or at least I
  couldn't find any use-case where this would matter at all. Anyways,
  it is still possible to do async shell in KDB, just don't see any
  need for this.

- Original FIQ debugger used a custom FIQ vector handling code, w/
  a lot of logic in it. In this approach I'm using the fact that
  FIQs are basically IRQs, except that we there are a bit more
  registers banked, and we can actually trap from the IRQ context.

  But this all does not prevent us from using a simple jump-table
  based approach as used in the generic ARM entry code. So, here
  I just reuse the generic approach.

Note that I testing the code on a modelled ARM machine (QEMU Versatile),
so there might be some issues on a real HW, but it works in QEMU tho. :-)

Assuming you have QEMU >= 1.1.0, you can easily play with the code
using ARM/versatile defconfig and command like this:

  qemu-system-arm -nographic -machine versatilepb \
  	-kernel linux/arch/arm/boot/zImage  \
  	-append "console=ttyAMA0 kgdboc=ttyAMA0 kgdb_fiq.enable=1"

TODO:

1. alignment_trap macro uses local label, so we have to put the label
   into each file that use the macro. We can get rid of the label;
2. Need per-machine kgdb_arch_enable_nmi(), probably will introduce
   a pointer to a func;
3. Since console interrupt is actually is overtaken by NMI handler, we
   should make serial/uart drivers stop using TX interrupts. This my
   homework to think how to do it better. Currently, we would just
   better not use console= and kgdboc= on the same tty (but it still
   works, just might cause troubles if you hit TX interrupt);
4. Address any comments. :-)

Thanks!

--
 arch/arm/Kconfig                            |   14 +++
 arch/arm/common/vic.c                       |   28 +++++
 arch/arm/include/asm/hardware/vic.h         |    2 +
 arch/arm/include/asm/kgdb.h                 |    8 ++
 arch/arm/kernel/Makefile                    |    1 +
 arch/arm/kernel/entry-armv.S                |  167 +-------------------------
 arch/arm/kernel/entry-header.S              |  170 +++++++++++++++++++++++++++
 arch/arm/kernel/kgdb_fiq.c                  |   78 ++++++++++++
 arch/arm/kernel/kgdb_fiq_entry.S            |   80 +++++++++++++
 arch/arm/mach-versatile/Makefile            |    1 +
 arch/arm/mach-versatile/include/mach/irqs.h |    1 +
 arch/arm/mach-versatile/kgdb_fiq.c          |   40 +++++++
 include/linux/kgdb.h                        |    9 ++
 kernel/debug/debug_core.c                   |   12 +-
 kernel/debug/kdb/kdb_debugger.c             |    4 +
 15 files changed, 448 insertions(+), 167 deletions(-)

p.s.

[1] Original Google's FIQ debugger, fiq_* files:
http://android.git.linaro.org/gitweb?p=kernel/common.git;a=tree;f=arch/arm/common;hb=refs/heads/android-3.4
And board support as an example of using it:
http://nv-tegra.nvidia.com/gitweb/?p=linux-2.6.git;a=commitdiff;h=461cb80c16e4e266ab6207a00767b59212148086

pp.s. If anyone curious, typical NMI entry looks like this
(I also executed a bit of commands):

Entering kdb (current=0xc781bd60, pid 1) due to NonMaskable Interrupt @ 0xc01510d0

Pid: 1, comm:              swapper
CPU: 0    Not tainted  (3.5.0-rc4+ #214)
PC is at __delay+0x0/0xc
LR is at panic+0x180/0x1b0
pc : [<c01510d0>]    lr : [<c0286b64>]    psr: 20000013
sp : c7823f24  ip : c7823f24  fp : c7823f38
r10: c02f35c4  r9 : 00000000  r8 : c0377988
r7 : 00000320  r6 : 000002bc  r5 : 00000040  r4 : 00000000
r3 : c0020f4c  r2 : 000002ce  r1 : ffffffff  r0 : 0000e2e1
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 00093177  Table: 00004000  DAC: 00000017
Backtrace:
[<c00173a4>] (dump_backtrace+0x0/0x10c) from [<c02867f4>] (dump_stack+0x18/0x1c)
 r6:0000000f r5:c0361d58 r4:c7823edc
[<c02867dc>] (dump_stack+0x0/0x1c) from [<c001506c>] (show_regs+0x44/0x50)
[<c0015028>] (show_regs+0x0/0x50) from [<c0287474>] (kdb_dumpregs+0x30/0x58)
 r4:c0383330
[<c0287444>] (kdb_dumpregs+0x0/0x58) from [<c00606e4>] (kdb_local.isra.5+0x354/0x5ec)
 r6:c0385534 r5:c7823edc r4:00000008
[<c0060390>] (kdb_local.isra.5+0x0/0x5ec) from [<c0060a28>] (kdb_main_loop+0xac/0x1bc)
[<c006097c>] (kdb_main_loop+0x0/0x1bc) from [<c0063020>] (kdb_stub+0x2e0/0x3e8)
 r8:c0385820 r7:c0364004 r6:c0382cb4 r5:c03857c8 r4:c7823e7c
[<c0062d40>] (kdb_stub+0x0/0x3e8) from [<c0059868>] (kgdb_cpu_enter.constprop.9+0x13c/0x4f8)
[<c005972c>] (kgdb_cpu_enter.constprop.9+0x0/0x4f8) from [<c0059f2c>] (kgdb_handle_exception+0x8c/0xa0)
[<c0059ea0>] (kgdb_handle_exception+0x0/0xa0) from [<c0008614>] (kgdb_fiq_do_handle+0x58/0x7c)
 r8:c0377988 r7:c7823f10 r6:ffffffff r5:c7823edc r4:c7822000
[<c00085bc>] (kgdb_fiq_do_handle+0x0/0x7c) from [<c0018df4>] (__fiq_svc+0x34/0x40)
Exception stack(0xc7823edc to 0xc7823f24)
3ec0:                                                                0000e2e1
3ee0: ffffffff 000002ce c0020f4c 00000000 00000040 000002bc 00000320 c0377988
3f00: 00000000 c02f35c4 c7823f38 c7823f24 c7823f24 c0286b64 c01510d0 20000013
3f20: ffffffff
 r5:20000013 r4:c01510d0
[<c02869e4>] (panic+0x0/0x1b0) from [<c0334d94>] (mount_block_root+0xe0/0x194)
 r3:00000000 r2:00000000 r1:c7823f50 r0:c02f355c
 r7:c789a000
[<c0334cb4>] (mount_block_root+0x0/0x194) from [<c0335030>] (mount_root+0xec/0x114)
[<c0334f44>] (mount_root+0x0/0x114) from [<c03351c0>] (prepare_namespace+0x168/0x1bc)
 r7:00000013 r6:c0025c0c r5:c0351b24 r4:c0377440
[<c0335058>] (prepare_namespace+0x0/0x1bc) from [<c03349e4>] (kernel_init+0xd0/0xfc)
 r5:c0351b24 r4:c0351b24
[<c0334914>] (kernel_init+0x0/0xfc) from [<c0025c0c>] (do_exit+0x0/0x2d8)
 r5:c0334914 r4:00000000
more>
kdb> md c01510d0
0xc01510d0 e2500001 8afffffd e1a0f00e e254c001   ..P...........T.
0xc01510e0 9a000033 e11c0004 0a000028 e1510004   3.......(.....Q.
0xc01510f0 e3a03000 3a00000b e16f2f14 e16fcf11   .0.....:./o...o.
0xc0151100 e042200c e3a0c001 e1a0c21c e1a02214   . B.........."..
0xc0151110 e1510002 2183300c 20511002 11b0c0ac   ..Q..0.!..Q ....
0xc0151120 e1a020a2 1afffff9 e3510000 e3a02000   . ........Q.. ..
0xc0151130 01500004 31a01000 31a0f00e e3a0c102   ..P....1...1....
0xc0151140 e1b00080 e0b11001 0a000005 31510004   ..............Q1
kdb> bp __delay
Instruction(i) BP #0 at 0xc01510d0 (__delay)
    is enabled  addr at 00000000c01510d0, hardtype=0 installed=0

kdb> go __delay

Entering kdb (current=0xc781bd60, pid 1) due to Breakpoint @ 0xc01510d0
kdb> bt
Stack traceback for pid 1
0xc781bd60        1        0  1    0   R  0xc781bf1c *swapper
Backtrace:
[<c00173a4>] (dump_backtrace+0x0/0x10c) from [<c0017804>] (show_stack+0x18/0x1c)
 r6:0000000f r5:c0361d58 r4:c0383330
[<c00177ec>] (show_stack+0x0/0x1c) from [<c006202c>] (kdb_show_stack+0x78/0x88)
[<c0061fb4>] (kdb_show_stack+0x0/0x88) from [<c00620c0>] (kdb_bt1.isra.0+0x84/0xd8)
 r8:00000032 r7:00000000 r6:00000000 r5:ffffffff r4:c781bd60
[<c006203c>] (kdb_bt1.isra.0+0x0/0xd8) from [<c00623b8>] (kdb_bt+0x2a4/0x348)
 r7:00000001 r6:00000000 r5:c03857d0 r4:c03856fc
[<c0062114>] (kdb_bt+0x0/0x348) from [<c005fdbc>] (kdb_parse+0x2cc/0x4f4)
 r8:00000032 r7:c03856fc r6:c02fa1f8 r5:c0383614 r4:00000009
[<c005faf0>] (kdb_parse+0x0/0x4f4) from [<c0060588>] (kdb_local.isra.5+0x1f8/0x5ec)
[<c0060390>] (kdb_local.isra.5+0x0/0x5ec) from [<c0060a28>] (kdb_main_loop+0xac/0x1bc)
[<c006097c>] (kdb_main_loop+0x0/0x1bc) from [<c0063020>] (kdb_stub+0x2e0/0x3e8)
 r8:c0385820 r7:c0364004 r6:c0382cb4 r5:c03857c8 r4:c7823de0
[<c0062d40>] (kdb_stub+0x0/0x3e8) from [<c0059868>] (kgdb_cpu_enter.constprop.9+0x13c/0x4f8)
[<c005972c>] (kgdb_cpu_enter.constprop.9+0x0/0x4f8) from [<c0059f2c>] (kgdb_handle_exception+0x8c/0xa0)
[<c0059ea0>] (kgdb_handle_exception+0x0/0xa0) from [<c0018ae0>] (kgdb_brk_fn+0x20/0x28)
 r8:c0377988 r7:00000000 r6:60000093 r5:c01510d0 r4:c7823edc
[<c0018ac0>] (kgdb_brk_fn+0x0/0x28) from [<c00084f0>] (do_undefinstr+0xdc/0x1a8)
[<c0008414>] (do_undefinstr+0x0/0x1a8) from [<c0013e1c>] (__und_svc+0x3c/0x60)
Exception stack(0xc7823edc to 0xc7823f24)
3ec0:                                                                0000e2e1
3ee0: ffffffff 000002ce c0020f4c 00000000 00000040 000002bc 00000320 c0377988
3f00: 00000000 c02f35c4 c7823f38 c7823f24 c7823f24 c0286b64 c01510d0 20000013
3f20: ffffffff
 r7:c7823f10 r6:ffffffff r5:20000013 r4:c01510d4
[<c02869e4>] (panic+0x0/0x1b0) from [<c0334d94>] (mount_block_root+0xe0/0x194)
 r3:00000000 r2:00000000 r1:c7823f50 r0:c02f355c
 r7:c789a000
[<c0334cb4>] (mount_block_root+0x0/0x194) from [<c0335030>] (mount_root+0xec/0x114)
[<c0334f44>] (mount_root+0x0/0x114) from [<c03351c0>] (prepare_namespace+0x168/0x1bc)
 r7:00000013 r6:c0025c0c r5:c0351b24 r4:c0377440
[<c0335058>] (prepare_namespace+0x0/0x1bc) from [<c03349e4>] (kernel_init+0xd0/0xfc)
 r5:c0351b24 r4:c0351b24
[<c0334914>] (kernel_init+0x0/0xfc) from [<c0025c0c>] (do_exit+0x0/0x2d8)
 r5:c0334914 r4:00000000
kdb>
kdb> ps
15 sleeping system daemon (state M) processes suppressed,
use 'ps A' to see all.
Task Addr       Pid   Parent [*] cpu State Thread     Command
0xc781bd60        1        0  1    0   R  0xc781bf1c *swapper

0xc781bd60        1        0  1    0   R  0xc781bf1c *swapper
0xc789dd60       13        2  0    0   R  0xc789df1c  kworker/0:1
0xc789d580       16        2  0    0   R  0xc789d73c  kworker/u:1
0xc796cd60       23        2  0    0   R  0xc796cf1c  deferwq

-- 
Anton Vorontsov
Email: cbouatmailru at gmail.com

[PATCH 1/6] kernel/debug: Make use of KGDB_REASON_NMI

[Anton Vorontsov]

Currently kernel never set KGDB_REASON_NMI. We do now, when we enter
KGDB/KDB from an NMI.

This is not to be confused with kgdb_nmicallback(), NMI callback is
an entry for the slave CPUs during CPUs roundup, but REASON_NMI is the
entry for the master CPU.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 kernel/debug/kdb/kdb_debugger.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/debug/kdb/kdb_debugger.c b/kernel/debug/kdb/kdb_debugger.c
index 8b68ce7..be7b33b 100644
--- a/kernel/debug/kdb/kdb_debugger.c
+++ b/kernel/debug/kdb/kdb_debugger.c
@@ -12,6 +12,7 @@
 #include <linux/kdb.h>
 #include <linux/kdebug.h>
 #include <linux/export.h>
+#include <linux/hardirq.h>
 #include "kdb_private.h"
 #include "../debug_core.h"
 
@@ -52,6 +53,9 @@ int kdb_stub(struct kgdb_state *ks)
 	if (atomic_read(&kgdb_setting_breakpoint))
 		reason = KDB_REASON_KEYBOARD;
 
+	if (in_nmi())
+		reason = KDB_REASON_NMI;
+
 	for (i = 0, bp = kdb_breakpoints; i < KDB_MAXBPT; i++, bp++) {
 		if ((bp->bp_enabled) && (bp->bp_addr == addr)) {
 			reason = KDB_REASON_BREAK;
-- 
1.7.10.4

[PATCH 2/6] kernel/debug: Mask KGDB NMI upon entry

[Anton Vorontsov]

The new arch callback should manage NMIs that usually cause KGDB to
enter. That is, not all NMIs should be enabled/disabled, but only
those that issue kgdb_handle_exception().

We must mask it as serial-line interrupt can be used as an NMI, so
if the original KGDB-entry cause was say a breakpoint, then every
input to KDB console will cause KGDB to reenter, which we don't want.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 include/linux/kgdb.h      |    9 +++++++++
 kernel/debug/debug_core.c |   12 +++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/include/linux/kgdb.h b/include/linux/kgdb.h
index c4d2fc1..47e739e 100644
--- a/include/linux/kgdb.h
+++ b/include/linux/kgdb.h
@@ -221,6 +221,15 @@ extern int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt);
  */
 extern void kgdb_arch_late(void);
 
+/**
+ *	kgdb_arch_enable_nmi - Enable or disable KGDB-entry NMI
+ *	@on: Flag to either enable or disable an NMI
+ *
+ *	This function manages NMIs that usually cause KGDB to enter. That is,
+ *	not all NMIs should be enabled or disabled, but only those that issue
+ *	kgdb_handle_exception().
+ */
+extern void kgdb_arch_enable_nmi(bool on);
 
 /**
  * struct kgdb_arch - Describe architecture specific values.
diff --git a/kernel/debug/debug_core.c b/kernel/debug/debug_core.c
index 0557f24..e285600 100644
--- a/kernel/debug/debug_core.c
+++ b/kernel/debug/debug_core.c
@@ -214,6 +214,10 @@ int __weak kgdb_skipexception(int exception, struct pt_regs *regs)
 	return 0;
 }
 
+void __weak kgdb_arch_enable_nmi(bool on)
+{
+}
+
 /*
  * Some architectures need cache flushes when we set/clear a
  * breakpoint:
@@ -672,6 +676,9 @@ kgdb_handle_exception(int evector, int signo, int ecode, struct pt_regs *regs)
 {
 	struct kgdb_state kgdb_var;
 	struct kgdb_state *ks = &kgdb_var;
+	int ret;
+
+	kgdb_arch_enable_nmi(0);
 
 	ks->cpu			= raw_smp_processor_id();
 	ks->ex_vector		= evector;
@@ -685,7 +692,10 @@ kgdb_handle_exception(int evector, int signo, int ecode, struct pt_regs *regs)
 	if (kgdb_info[ks->cpu].enter_kgdb != 0)
 		return 0;
 
-	return kgdb_cpu_enter(ks, regs, DCPU_WANT_MASTER);
+	ret = kgdb_cpu_enter(ks, regs, DCPU_WANT_MASTER);
+
+	kgdb_arch_enable_nmi(1);
+	return ret;
 }
 
 int kgdb_nmicallback(int cpu, void *regs)
-- 
1.7.10.4

[PATCH 3/6] ARM: Move some macros from entry-armv to entry-header

[Anton Vorontsov]

Just move the macros into header file as we would want to use them for
KGDB FIQ entry code.

The following macros were moved:

 - svc_entry
 - usr_entry
 - kuser_cmpxchg_check
 - vector_stub

To make kuser_cmpxchg_check actually work across different files, we
also have to make kuser_cmpxchg64_fixup global.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 arch/arm/kernel/entry-armv.S   |  167 +--------------------------------------
 arch/arm/kernel/entry-header.S |  170 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 171 insertions(+), 166 deletions(-)

diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index 0d1851c..6aeb9b8 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -136,57 +136,6 @@ common_invalid:
 	b	bad_mode
 ENDPROC(__und_invalid)
 
-/*
- * SVC mode handlers
- */
-
-#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5)
-#define SPFIX(code...) code
-#else
-#define SPFIX(code...)
-#endif
-
-	.macro	svc_entry, stack_hole=0
- UNWIND(.fnstart		)
- UNWIND(.save {r0 - pc}		)
-	sub	sp, sp, #(S_FRAME_SIZE + \stack_hole - 4)
-#ifdef CONFIG_THUMB2_KERNEL
- SPFIX(	str	r0, [sp]	)	@ temporarily saved
- SPFIX(	mov	r0, sp		)
- SPFIX(	tst	r0, #4		)	@ test original stack alignment
- SPFIX(	ldr	r0, [sp]	)	@ restored
-#else
- SPFIX(	tst	sp, #4		)
-#endif
- SPFIX(	subeq	sp, sp, #4	)
-	stmia	sp, {r1 - r12}
-
-	ldmia	r0, {r3 - r5}
-	add	r7, sp, #S_SP - 4	@ here for interlock avoidance
-	mov	r6, #-1			@  ""  ""      ""       ""
-	add	r2, sp, #(S_FRAME_SIZE + \stack_hole - 4)
- SPFIX(	addeq	r2, r2, #4	)
-	str	r3, [sp, #-4]!		@ save the "real" r0 copied
-					@ from the exception stack
-
-	mov	r3, lr
-
-	@
-	@ We are now ready to fill in the remaining blanks on the stack:
-	@
-	@  r2 - sp_svc
-	@  r3 - lr_svc
-	@  r4 - lr_<exception>, already fixed up for correct return/restart
-	@  r5 - spsr_<exception>
-	@  r6 - orig_r0 (see pt_regs definition in ptrace.h)
-	@
-	stmia	r7, {r2 - r6}
-
-#ifdef CONFIG_TRACE_IRQFLAGS
-	bl	trace_hardirqs_off
-#endif
-	.endm
-
 	.align	5
 __dabt_svc:
 	svc_entry
@@ -328,71 +277,8 @@ ENDPROC(__pabt_svc)
 
 /*
  * User mode handlers
- *
- * EABI note: sp_svc is always 64-bit aligned here, so should S_FRAME_SIZE
  */
 
-#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5) && (S_FRAME_SIZE & 7)
-#error "sizeof(struct pt_regs) must be a multiple of 8"
-#endif
-
-	.macro	usr_entry
- UNWIND(.fnstart	)
- UNWIND(.cantunwind	)	@ don't unwind the user space
-	sub	sp, sp, #S_FRAME_SIZE
- ARM(	stmib	sp, {r1 - r12}	)
- THUMB(	stmia	sp, {r0 - r12}	)
-
-	ldmia	r0, {r3 - r5}
-	add	r0, sp, #S_PC		@ here for interlock avoidance
-	mov	r6, #-1			@  ""  ""     ""        ""
-
-	str	r3, [sp]		@ save the "real" r0 copied
-					@ from the exception stack
-
-	@
-	@ We are now ready to fill in the remaining blanks on the stack:
-	@
-	@  r4 - lr_<exception>, already fixed up for correct return/restart
-	@  r5 - spsr_<exception>
-	@  r6 - orig_r0 (see pt_regs definition in ptrace.h)
-	@
-	@ Also, separately save sp_usr and lr_usr
-	@
-	stmia	r0, {r4 - r6}
- ARM(	stmdb	r0, {sp, lr}^			)
- THUMB(	store_user_sp_lr r0, r1, S_SP - S_PC	)
-
-	@
-	@ Enable the alignment trap while in kernel mode
-	@
-	alignment_trap r0
-
-	@
-	@ Clear FP to mark the first stack frame
-	@
-	zero_fp
-
-#ifdef CONFIG_IRQSOFF_TRACER
-	bl	trace_hardirqs_off
-#endif
-	.endm
-
-	.macro	kuser_cmpxchg_check
-#if !defined(CONFIG_CPU_32v6K) && !defined(CONFIG_NEEDS_SYSCALL_FOR_CMPXCHG)
-#ifndef CONFIG_MMU
-#warning "NPTL on non MMU needs fixing"
-#else
-	@ Make sure our user space atomic helper is restarted
-	@ if it was interrupted in a critical region.  Here we
-	@ perform a quick test inline since it should be false
-	@ 99.9999% of the time.  The rest is done out of line.
-	cmp	r4, #TASK_SIZE
-	blhs	kuser_cmpxchg64_fixup
-#endif
-#endif
-	.endm
-
 	.align	5
 __dabt_usr:
 	usr_entry
@@ -813,6 +699,7 @@ __kuser_cmpxchg64:				@ 0xffff0f60
 	ldmfd	sp!, {r4, r5, r6, pc}
 
 	.text
+	.global kuser_cmpxchg64_fixup
 kuser_cmpxchg64_fixup:
 	@ Called from kuser_cmpxchg_fixup.
 	@ r4 = address of interrupted insn (must be preserved).
@@ -943,58 +830,6 @@ __kuser_helper_end:
 
  THUMB(	.thumb	)
 
-/*
- * Vector stubs.
- *
- * This code is copied to 0xffff0200 so we can use branches in the
- * vectors, rather than ldr's.  Note that this code must not
- * exceed 0x300 bytes.
- *
- * Common stub entry macro:
- *   Enter in IRQ mode, spsr = SVC/USR CPSR, lr = SVC/USR PC
- *
- * SP points to a minimal amount of processor-private memory, the address
- * of which is copied into r0 for the mode specific abort handler.
- */
-	.macro	vector_stub, name, mode, correction=0
-	.align	5
-
-vector_\name:
-	.if \correction
-	sub	lr, lr, #\correction
-	.endif
-
-	@
-	@ Save r0, lr_<exception> (parent PC) and spsr_<exception>
-	@ (parent CPSR)
-	@
-	stmia	sp, {r0, lr}		@ save r0, lr
-	mrs	lr, spsr
-	str	lr, [sp, #8]		@ save spsr
-
-	@
-	@ Prepare for SVC32 mode.  IRQs remain disabled.
-	@
-	mrs	r0, cpsr
-	eor	r0, r0, #(\mode ^ SVC_MODE | PSR_ISETSTATE)
-	msr	spsr_cxsf, r0
-
-	@
-	@ the branch table must immediately follow this code
-	@
-	and	lr, lr, #0x0f
- THUMB(	adr	r0, 1f			)
- THUMB(	ldr	lr, [r0, lr, lsl #2]	)
-	mov	r0, sp
- ARM(	ldr	lr, [pc, lr, lsl #2]	)
-	movs	pc, lr			@ branch to handler in SVC mode
-ENDPROC(vector_\name)
-
-	.align	2
-	@ handler addresses follow this label
-1:
-	.endm
-
 	.globl	__stubs_start
 __stubs_start:
 /*
diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S
index 9a8531e..c3c09ac 100644
--- a/arch/arm/kernel/entry-header.S
+++ b/arch/arm/kernel/entry-header.S
@@ -73,6 +73,109 @@
 	msr	cpsr_c, \rtemp			@ switch back to the SVC mode
 	.endm
 
+/*
+ * Vector stubs.
+ *
+ * This code is copied to 0xffff0200 so we can use branches in the
+ * vectors, rather than ldr's.  Note that this code must not
+ * exceed 0x300 bytes.
+ *
+ * Common stub entry macro:
+ *   Enter in IRQ mode, spsr = SVC/USR CPSR, lr = SVC/USR PC
+ *
+ * SP points to a minimal amount of processor-private memory, the address
+ * of which is copied into r0 for the mode specific abort handler.
+ */
+	.macro	vector_stub, name, mode, correction=0
+	.align	5
+
+vector_\name:
+	.if \correction
+	sub	lr, lr, #\correction
+	.endif
+
+	@
+	@ Save r0, lr_<exception> (parent PC) and spsr_<exception>
+	@ (parent CPSR)
+	@
+	stmia	sp, {r0, lr}		@ save r0, lr
+	mrs	lr, spsr
+	str	lr, [sp, #8]		@ save spsr
+
+	@
+	@ Prepare for SVC32 mode.  IRQs remain disabled.
+	@
+	mrs	r0, cpsr
+	eor	r0, r0, #(\mode ^ SVC_MODE | PSR_ISETSTATE)
+	msr	spsr_cxsf, r0
+
+	@
+	@ the branch table must immediately follow this code
+	@
+	and	lr, lr, #0x0f
+ THUMB(	adr	r0, 1f			)
+ THUMB(	ldr	lr, [r0, lr, lsl #2]	)
+	mov	r0, sp
+ ARM(	ldr	lr, [pc, lr, lsl #2]	)
+	movs	pc, lr			@ branch to handler in SVC mode
+ENDPROC(vector_\name)
+
+	.align	2
+	@ handler addresses follow this label
+1:
+	.endm
+
+/*
+ * SVC mode handlers
+ */
+
+#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5)
+#define SPFIX(code...) code
+#else
+#define SPFIX(code...)
+#endif
+
+	.macro	svc_entry, stack_hole=0
+ UNWIND(.fnstart		)
+ UNWIND(.save {r0 - pc}		)
+	sub	sp, sp, #(S_FRAME_SIZE + \stack_hole - 4)
+#ifdef CONFIG_THUMB2_KERNEL
+ SPFIX(	str	r0, [sp]	)	@ temporarily saved
+ SPFIX(	mov	r0, sp		)
+ SPFIX(	tst	r0, #4		)	@ test original stack alignment
+ SPFIX(	ldr	r0, [sp]	)	@ restored
+#else
+ SPFIX(	tst	sp, #4		)
+#endif
+ SPFIX(	subeq	sp, sp, #4	)
+	stmia	sp, {r1 - r12}
+
+	ldmia	r0, {r3 - r5}
+	add	r7, sp, #S_SP - 4	@ here for interlock avoidance
+	mov	r6, #-1			@  ""  ""      ""       ""
+	add	r2, sp, #(S_FRAME_SIZE + \stack_hole - 4)
+ SPFIX(	addeq	r2, r2, #4	)
+	str	r3, [sp, #-4]!		@ save the "real" r0 copied
+					@ from the exception stack
+
+	mov	r3, lr
+
+	@
+	@ We are now ready to fill in the remaining blanks on the stack:
+	@
+	@  r2 - sp_svc
+	@  r3 - lr_svc
+	@  r4 - lr_<exception>, already fixed up for correct return/restart
+	@  r5 - spsr_<exception>
+	@  r6 - orig_r0 (see pt_regs definition in ptrace.h)
+	@
+	stmia	r7, {r2 - r6}
+
+#ifdef CONFIG_TRACE_IRQFLAGS
+	bl	trace_hardirqs_off
+#endif
+	.endm
+
 #ifndef CONFIG_THUMB2_KERNEL
 	.macro	svc_exit, rpsr
 	msr	spsr_cxsf, \rpsr
@@ -164,6 +267,73 @@
 #endif	/* !CONFIG_THUMB2_KERNEL */
 
 /*
+ * User mode handlers
+ *
+ * EABI note: sp_svc is always 64-bit aligned here, so should S_FRAME_SIZE
+ */
+
+#if defined(CONFIG_AEABI) && (__LINUX_ARM_ARCH__ >= 5) && (S_FRAME_SIZE & 7)
+#error "sizeof(struct pt_regs) must be a multiple of 8"
+#endif
+
+	.macro	usr_entry
+ UNWIND(.fnstart	)
+ UNWIND(.cantunwind	)	@ don't unwind the user space
+	sub	sp, sp, #S_FRAME_SIZE
+ ARM(	stmib	sp, {r1 - r12}	)
+ THUMB(	stmia	sp, {r0 - r12}	)
+
+	ldmia	r0, {r3 - r5}
+	add	r0, sp, #S_PC		@ here for interlock avoidance
+	mov	r6, #-1			@  ""  ""     ""        ""
+
+	str	r3, [sp]		@ save the "real" r0 copied
+					@ from the exception stack
+
+	@
+	@ We are now ready to fill in the remaining blanks on the stack:
+	@
+	@  r4 - lr_<exception>, already fixed up for correct return/restart
+	@  r5 - spsr_<exception>
+	@  r6 - orig_r0 (see pt_regs definition in ptrace.h)
+	@
+	@ Also, separately save sp_usr and lr_usr
+	@
+	stmia	r0, {r4 - r6}
+ ARM(	stmdb	r0, {sp, lr}^			)
+ THUMB(	store_user_sp_lr r0, r1, S_SP - S_PC	)
+
+	@
+	@ Enable the alignment trap while in kernel mode
+	@
+	alignment_trap r0
+
+	@
+	@ Clear FP to mark the first stack frame
+	@
+	zero_fp
+
+#ifdef CONFIG_IRQSOFF_TRACER
+	bl	trace_hardirqs_off
+#endif
+	.endm
+
+	.macro	kuser_cmpxchg_check
+#if !defined(CONFIG_CPU_32v6K) && !defined(CONFIG_NEEDS_SYSCALL_FOR_CMPXCHG)
+#ifndef CONFIG_MMU
+#warning "NPTL on non MMU needs fixing"
+#else
+	@ Make sure our user space atomic helper is restarted
+	@ if it was interrupted in a critical region.  Here we
+	@ perform a quick test inline since it should be false
+	@ 99.9999% of the time.  The rest is done out of line.
+	cmp	r4, #TASK_SIZE
+	blhs	kuser_cmpxchg64_fixup
+#endif
+#endif
+	.endm
+
+/*
  * These are the registers used in the syscall handler, and allow us to
  * have in theory up to 7 arguments to a function - r0 to r6.
  *
-- 
1.7.10.4

[PATCH 4/6] ARM: Add KGDB/KDB FIQ debugger generic code

[Anton Vorontsov]

The FIQ debugger may be used to debug situations when the kernel stuck
in uninterruptable sections, e.g. the kernel infinitely loops or
deadlocked in an interrupt or with interrupts disabled.

By default KGDB FIQ is disabled in runtime, but can be enabled with
kgdb_fiq.enable=1 kernel command line option.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 arch/arm/Kconfig                 |   14 +++++++
 arch/arm/include/asm/kgdb.h      |    8 ++++
 arch/arm/kernel/Makefile         |    1 +
 arch/arm/kernel/kgdb_fiq.c       |   78 +++++++++++++++++++++++++++++++++++++
 arch/arm/kernel/kgdb_fiq_entry.S |   80 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 181 insertions(+)
 create mode 100644 arch/arm/kernel/kgdb_fiq.c
 create mode 100644 arch/arm/kernel/kgdb_fiq_entry.S

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a91009c..bc558d1 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -172,6 +172,20 @@ config GENERIC_ISA_DMA
 config FIQ
 	bool
 
+config KGDB_FIQ
+	bool "KGDB/KDB FIQ debugger"
+	select FIQ
+	help
+	  The FIQ debugger may be used to debug situations when the
+	  kernel stuck in uninterruptable sections, e.g. the kernel
+	  infinitely loops or deadlocked in an interrupt or with
+	  interrupts disabled.
+
+	  By default KGDB FIQ is disabled in runtime, but can be
+	  enabled with kgdb_fiq.enable=1 kernel command line option.
+
+	  If unsure, say N.
+
 config NEED_RET_TO_USER
 	bool
 
diff --git a/arch/arm/include/asm/kgdb.h b/arch/arm/include/asm/kgdb.h
index 48066ce..ee51024 100644
--- a/arch/arm/include/asm/kgdb.h
+++ b/arch/arm/include/asm/kgdb.h
@@ -11,6 +11,8 @@
 #define __ARM_KGDB_H__
 
 #include <linux/ptrace.h>
+#include <linux/linkage.h>
+#include <asm/exception.h>
 
 /*
  * GDB assumes that we're a user process being debugged, so
@@ -47,6 +49,12 @@ static inline void arch_kgdb_breakpoint(void)
 extern void kgdb_handle_bus_error(void);
 extern int kgdb_fault_expected;
 
+extern char kgdb_fiq_handler;
+extern char kgdb_fiq_handler_end;
+asmlinkage void __exception_irq_entry kgdb_fiq_do_handle(struct pt_regs *regs);
+typedef bool (*is_kgdb_nmi_fn)(void);
+extern void kgdb_register_nmi(is_kgdb_nmi_fn is_kgdb_nmi);
+
 #endif /* !__ASSEMBLY__ */
 
 /*
diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile
index 7ad2d5c..5aa079b 100644
--- a/arch/arm/kernel/Makefile
+++ b/arch/arm/kernel/Makefile
@@ -56,6 +56,7 @@ obj-$(CONFIG_ATAGS_PROC)	+= atags.o
 obj-$(CONFIG_OABI_COMPAT)	+= sys_oabi-compat.o
 obj-$(CONFIG_ARM_THUMBEE)	+= thumbee.o
 obj-$(CONFIG_KGDB)		+= kgdb.o
+obj-$(CONFIG_KGDB_FIQ)		+= kgdb_fiq_entry.o kgdb_fiq.o
 obj-$(CONFIG_ARM_UNWIND)	+= unwind.o
 obj-$(CONFIG_HAVE_TCM)		+= tcm.o
 obj-$(CONFIG_OF)		+= devtree.o
diff --git a/arch/arm/kernel/kgdb_fiq.c b/arch/arm/kernel/kgdb_fiq.c
new file mode 100644
index 0000000..287504d
--- /dev/null
+++ b/arch/arm/kernel/kgdb_fiq.c
@@ -0,0 +1,78 @@
+/*
+ * KGDB FIQ entry
+ *
+ * Copyright 2010 Google, Inc.
+ *		  Arve Hj?nnev?g <arve@android.com>
+ *		  Colin Cross <ccross@android.com>
+ * Copyright 2012 Linaro Ltd.
+ *		  Anton Vorontsov <anton.vorontsov@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/hardirq.h>
+#include <linux/kgdb.h>
+#include <asm/fiq.h>
+#include <asm/exception.h>
+
+static int kgdb_fiq_enabled;
+module_param_named(enable, kgdb_fiq_enabled, int, 0600);
+MODULE_PARM_DESC(enable, "set to 1 to enable FIQ KGDB");
+
+static long kgdb_fiq_setup_stack(void *info)
+{
+	struct pt_regs regs;
+
+	regs.ARM_sp = __get_free_pages(GFP_KERNEL, THREAD_SIZE_ORDER) +
+			THREAD_START_SP;
+	WARN_ON(!regs.ARM_sp);
+
+	set_fiq_regs(&regs);
+	return 0;
+}
+
+static is_kgdb_nmi_fn is_kgdb_nmi;
+
+asmlinkage void __exception_irq_entry kgdb_fiq_do_handle(struct pt_regs *regs)
+{
+	if (!is_kgdb_nmi())
+		return;
+
+	nmi_enter();
+	kgdb_handle_exception(1, 0, 0, regs);
+	nmi_exit();
+}
+
+static struct fiq_handler kgdb_fiq_desc = {
+	.name = "kgdb",
+};
+
+void kgdb_register_nmi(is_kgdb_nmi_fn _is_kgdb_nmi)
+{
+	int err;
+	int cpu;
+
+	if (!kgdb_fiq_enabled)
+		return;
+
+	is_kgdb_nmi = _is_kgdb_nmi;
+
+	err = claim_fiq(&kgdb_fiq_desc);
+	if (err) {
+		WARN_ON(1);
+		return;
+	}
+
+	for_each_possible_cpu(cpu)
+		work_on_cpu(cpu, kgdb_fiq_setup_stack, NULL);
+
+	set_fiq_handler(&kgdb_fiq_handler,
+			&kgdb_fiq_handler_end - &kgdb_fiq_handler);
+
+	kgdb_arch_enable_nmi(1);
+}
diff --git a/arch/arm/kernel/kgdb_fiq_entry.S b/arch/arm/kernel/kgdb_fiq_entry.S
new file mode 100644
index 0000000..8524671
--- /dev/null
+++ b/arch/arm/kernel/kgdb_fiq_entry.S
@@ -0,0 +1,80 @@
+/*
+ * KGDB FIQ entry
+ *
+ * Copyright 1996,1997,1998 Russell King.
+ * Copyright 2012 Linaro Ltd.
+ *		  Anton Vorontsov <anton.vorontsov@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/memory.h>
+#include <asm/unwind.h>
+#include "entry-header.S"
+
+	.text
+
+	.macro	fiq_handler
+	ldr	r1, =.LCdohandle
+	mov	r0, sp
+	adr	lr, BSYM(9997f)
+	ldr	pc, [r1]
+9997:
+	.endm
+
+	.align	5
+__fiq_svc:
+	svc_entry
+	fiq_handler
+	svc_exit r5				@ return from exception
+ UNWIND(.fnend		)
+ENDPROC(__fiq_svc)
+	.ltorg
+
+	.align	5
+__fiq_usr:
+	usr_entry
+	kuser_cmpxchg_check
+	fiq_handler
+	get_thread_info tsk
+	mov	why, #0
+	b	ret_to_user_from_irq
+ UNWIND(.fnend		)
+ENDPROC(__fiq_usr)
+	.ltorg
+	@ This is needed for kuser_cmpxchg_check/alignment_trap
+	.global cr_alignment
+.LCcralign:
+	.word cr_alignment
+
+.LCdohandle:
+	.long kgdb_fiq_do_handle
+
+	.global kgdb_fiq_handler
+kgdb_fiq_handler:
+
+	vector_stub	fiq, FIQ_MODE, 4
+
+	.long	__fiq_usr			@  0  (USR_26 / USR_32)
+	.long	__fiq_svc			@  1  (FIQ_26 / FIQ_32)
+	.long	__fiq_svc			@  2  (IRQ_26 / IRQ_32)
+	.long	__fiq_svc			@  3  (SVC_26 / SVC_32)
+	.long	__fiq_svc			@  4
+	.long	__fiq_svc			@  5
+	.long	__fiq_svc			@  6
+	.long	__fiq_svc			@  7
+	.long	__fiq_svc			@  8
+	.long	__fiq_svc			@  9
+	.long	__fiq_svc			@  a
+	.long	__fiq_svc			@  b
+	.long	__fiq_svc			@  c
+	.long	__fiq_svc			@  d
+	.long	__fiq_svc			@  e
+	.long	__fiq_svc			@  f
+
+	.global kgdb_fiq_handler_end
+kgdb_fiq_handler_end:
-- 
1.7.10.4

[PATCH 5/6] ARM: VIC: Add a couple of low-level FIQ management helpers

[Anton Vorontsov]

Just a couple of calls to manage VIC FIQ routing. We'll use them for
KGDB FIQ support on ARM Versatile machines.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 arch/arm/common/vic.c               |   28 ++++++++++++++++++++++++++++
 arch/arm/include/asm/hardware/vic.h |    2 ++
 2 files changed, 30 insertions(+)

diff --git a/arch/arm/common/vic.c b/arch/arm/common/vic.c
index e0d5388..df2fc82 100644
--- a/arch/arm/common/vic.c
+++ b/arch/arm/common/vic.c
@@ -66,6 +66,34 @@ static struct vic_device vic_devices[CONFIG_ARM_VIC_NR];
 
 static int vic_id;
 
+static void __iomem *vic_base(struct irq_data *d)
+{
+	return (void __iomem *)irq_data_get_irq_chip_data(d);
+}
+
+void vic_fiq_select(unsigned int irq, bool on)
+{
+	void __iomem *base = vic_base(&irq_to_desc(irq)->irq_data);
+	void __iomem *sel = base + VIC_INT_SELECT;
+	u32 msk = 1 << irq;
+	u32 val;
+
+	pr_debug("rerouting VIC vector %d to %s\n", irq, on ? "FIQ" : "IRQ");
+
+	val = readl(sel);
+	val &= ~msk;
+	if (on)
+		val |= msk;
+	writel(val, sel);
+}
+
+bool vic_is_fiq_rised(unsigned int irq)
+{
+	void __iomem *base = vic_base(&irq_to_desc(irq)->irq_data);
+
+	return readl(base + VIC_FIQ_STATUS) & (1 << irq);
+}
+
 /**
  * vic_init2 - common initialisation code
  * @base: Base of the VIC.
diff --git a/arch/arm/include/asm/hardware/vic.h b/arch/arm/include/asm/hardware/vic.h
index e14af1a..2728975 100644
--- a/arch/arm/include/asm/hardware/vic.h
+++ b/arch/arm/include/asm/hardware/vic.h
@@ -52,6 +52,8 @@ void __vic_init(void __iomem *base, unsigned int irq_start, u32 vic_sources,
 void vic_init(void __iomem *base, unsigned int irq_start, u32 vic_sources, u32 resume_sources);
 int vic_of_init(struct device_node *node, struct device_node *parent);
 void vic_handle_irq(struct pt_regs *regs);
+void vic_fiq_select(unsigned int irq, bool on);
+bool vic_is_fiq_rised(unsigned int irq);
 
 #endif /* __ASSEMBLY__ */
 #endif
-- 
1.7.10.4

[PATCH 6/6] ARM: versatile: Make able to use UART ports for KGDB FIQ debugger

[Anton Vorontsov]

If enabled, kernel will able to enter KGDB upon serial line activity on
UART ports.

Note that even with this patch and CONFIG_KGDB_FIQ is enabled, you still
need to pass kgdb_fiq.enable=1 kernel command line option, otherwise UART
will behave in a normal way.

By default UART0 is used, but this can be changed via kgdb_fiq.uart_num
kernel command line option.

Signed-off-by: Anton Vorontsov <anton.vorontsov@linaro.org>
---
 arch/arm/mach-versatile/Makefile            |    1 +
 arch/arm/mach-versatile/include/mach/irqs.h |    1 +
 arch/arm/mach-versatile/kgdb_fiq.c          |   40 +++++++++++++++++++++++++++
 3 files changed, 42 insertions(+)
 create mode 100644 arch/arm/mach-versatile/kgdb_fiq.c

diff --git a/arch/arm/mach-versatile/Makefile b/arch/arm/mach-versatile/Makefile
index 81fa3fe..bfd761f 100644
--- a/arch/arm/mach-versatile/Makefile
+++ b/arch/arm/mach-versatile/Makefile
@@ -7,3 +7,4 @@ obj-$(CONFIG_ARCH_VERSATILE_PB)		+= versatile_pb.o
 obj-$(CONFIG_MACH_VERSATILE_AB)		+= versatile_ab.o
 obj-$(CONFIG_MACH_VERSATILE_DT)		+= versatile_dt.o
 obj-$(CONFIG_PCI)			+= pci.o
+obj-$(CONFIG_KGDB_FIQ)			+= kgdb_fiq.o
diff --git a/arch/arm/mach-versatile/include/mach/irqs.h b/arch/arm/mach-versatile/include/mach/irqs.h
index bf44c61..fcd2a95 100644
--- a/arch/arm/mach-versatile/include/mach/irqs.h
+++ b/arch/arm/mach-versatile/include/mach/irqs.h
@@ -26,6 +26,7 @@
  *  held within platform.h
  */
 #define IRQ_VIC_START		0
+#define FIQ_START		IRQ_VIC_START
 #define IRQ_WDOGINT		(IRQ_VIC_START + INT_WDOGINT)
 #define IRQ_SOFTINT		(IRQ_VIC_START + INT_SOFTINT)
 #define IRQ_COMMRx		(IRQ_VIC_START + INT_COMMRx)
diff --git a/arch/arm/mach-versatile/kgdb_fiq.c b/arch/arm/mach-versatile/kgdb_fiq.c
new file mode 100644
index 0000000..c82e8ce
--- /dev/null
+++ b/arch/arm/mach-versatile/kgdb_fiq.c
@@ -0,0 +1,40 @@
+/*
+ * KGDB FIQ entry board/IC support
+ *
+ * Copyright 2012 Linaro Ltd.
+ *		  Anton Vorontsov <anton.vorontsov@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/kgdb.h>
+#include <mach/hardware.h>
+#include <mach/platform.h>
+#include <asm/kgdb.h>
+#include <asm/hardware/vic.h>
+
+static int kgdb_fiq;
+module_param_named(uart_num, kgdb_fiq, int, 0600);
+MODULE_PARM_DESC(uart_num, "UART<number> port to use for KGDB FIQ");
+
+static bool is_kgdb_nmi(void)
+{
+	return vic_is_fiq_rised(kgdb_fiq);
+}
+
+void kgdb_arch_enable_nmi(bool on)
+{
+	vic_fiq_select(kgdb_fiq, on);
+}
+
+static int kgdb_fiq_init(void)
+{
+	kgdb_fiq += INT_UARTINT0;
+	kgdb_register_nmi(is_kgdb_nmi);
+	return 0;
+}
+console_initcall(kgdb_fiq_init);
-- 
1.7.10.4

[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[Colin Cross]

On Thu, Jul 5, 2012 at 4:10 PM, Anton Vorontsov
<anton.vorontsov@linaro.org> wrote:
>
> Hi all,
>
> These patches introduce KGDB FIQ debugger support. The idea (and some
> code, of course) comes from Google's FIQ debugger[1]. There are some
> differences (mostly implementation details, feature-wise they're almost
> equivalent, or can be made equivalent, if desired).

I haven't looked at this in detail yet, but the concept it sounds
pretty nice.  One difference I would like to highlight is that the FIQ
debugger is designed to be limited enough to leave enabled on
production devices, and is often the only debugging feature available
on just-before-production devices when trying to nail those last few
pesky bugs.  KGDB can obviously only be enabled on development
devices, although perhaps a more limited KDB could be left enabled.

> The FIQ debugger is a facility that can be used to debug situations
> when the kernel stuck in uninterruptable sections, e.g. the kernel
> infinitely loops or deadlocked in an interrupt or with interrupts
> disabled. On some development boards there is even a special NMI
> button, which is very useful for debugging weird kernel hangs.
>
> And FIQ is basically an NMI, it has a higher priority than IRQs, and
> upon IRQ exception FIQs are not disabled. It is still possible to
> disable FIQs (as well as some "NMIs" on other architectures), but via
> special means.
>
> So, here FIQs and NMIs are synonyms, but in the code I use NMI term
> for arch-independent code, and FIQs for ARM code.

Unfortunately, FIQs have been repurposed as secure interrupts on every
ARM SoC that supports TrustZone, which is basically all of the latest
generation, as well as a few of the previous generation.  When an FIQ
arrives, the cpu traps into the secure mode, generally running a
separate secure OS written by a 3rd party vendor.  We've tried to get
some SoC secure implementations to drop out of secure mode and into
the FIQ exception vector for specific irqs, with the registers set up
to allow the FIQ handler to return back to the original execution
point, but it's been successful.  It's a tricky problem, because
non-secure can't enable/disable FIQs, so they have to be enabled when
secure jumps to the FIQ exception vector, which then causes reentrancy
issues.

An alternate solution could be to standardize a secure FIQ handler
interface, where the secure side could drop into the kernel with the
registers already saved somewhere, and then the kernel would trap back
into secure mode when the handler finished to allow the secure
interrupt to be re-enabled.  Obviously not part of this patch set, but
until something like that exists, this isn't going to get used on
today's SoCs.

> A few years ago KDB wasn't yet ready for production, or even not
> well-known, so originally Google implemented its own FIQ debugger
> that included its own shell, ring-buffer, commands, dumping,
> backtracing logic and whatnot. This is very much like PowerPC's xmon
> (arch/powerpc/xmon), except that xmon was there for a decade, so it
> even predates KDB.
>
> Anyway, nowadays KGDB/KDB is the cross-platform debugger, and the
> only feature that was missing is NMI handling. This is now fixed for
> ARM.
>
> There a few differences comparing to the original (Google's) FIQ
> debugger:
>
> - Doing stuff in FIQ context is dangerous, as there we are not allowed
>   to cause aborts or faults. In the original FIQ debugger there was a
>   "signal" software-induced interrupt, upon exit from FIQ it would fire,
>   and we would continue to execute "dangerous" commands from there.
>
>   In KGDB/KDB we don't use signal interrupts. We can do easier:
>   set up a breakpoint, continue, and you'll trap into KGDB again
>   in a safe context.

I like this, much better than generating an interrupt (which may not
even be possible on some platforms).

>   It works for most cases, but I can imagine cases when you can't
>   set up a breakpoint. For these cases we'd better introduce a
>   KDB command "exit_nmi", that will rise the SW IRQ, after which
>   we're allowed to do anything.
>
> - KGDB/KDB FIQ debugger shell is synchronous. In Google's version
>   you could have a dedicated shell always running in the FIQ context,
>   so when you type something on a serial line, you won't actually cause
>   any debugging actions, FIQ would save the characters in its own
>   buffer and continue execution normally. But when you hit return key
>   after the command, then the command is executed.
>
>   In KGDB/KDB FIQ debugger it is different. When you start any activity
>   on the FIQ-enabled serial console, you'll enter KGDB and kernel will
>   stop until you instruct it to continue.
>
>   This might look as a drastic change, but it is not. There is actually
>   no difference whether you have sync or async shell, or at least I
>   couldn't find any use-case where this would matter at all. Anyways,
>   it is still possible to do async shell in KDB, just don't see any
>   need for this.

I think it could be an issue if KDB stopped execution whenever it
received any character.  Serial ports are often noisy, especially when
muxed over another port (we often use serial over the headset
connector).  Noise on the async command line just causes characters
that are ignored, on a command line that blocked execution noise would
be catastrophic.

> - Original FIQ debugger used a custom FIQ vector handling code, w/
>   a lot of logic in it. In this approach I'm using the fact that
>   FIQs are basically IRQs, except that we there are a bit more
>   registers banked, and we can actually trap from the IRQ context.
>
>   But this all does not prevent us from using a simple jump-table
>   based approach as used in the generic ARM entry code. So, here
>   I just reuse the generic approach.
>
> Note that I testing the code on a modelled ARM machine (QEMU Versatile),
> so there might be some issues on a real HW, but it works in QEMU tho. :-)
>
> Assuming you have QEMU >= 1.1.0, you can easily play with the code
> using ARM/versatile defconfig and command like this:
>
>   qemu-system-arm -nographic -machine versatilepb \
>         -kernel linux/arch/arm/boot/zImage  \
>         -append "console=ttyAMA0 kgdboc=ttyAMA0 kgdb_fiq.enable=1"
>
> TODO:
>
> 1. alignment_trap macro uses local label, so we have to put the label
>    into each file that use the macro. We can get rid of the label;
> 2. Need per-machine kgdb_arch_enable_nmi(), probably will introduce
>    a pointer to a func;
> 3. Since console interrupt is actually is overtaken by NMI handler, we
>    should make serial/uart drivers stop using TX interrupts. This my
>    homework to think how to do it better. Currently, we would just
>    better not use console= and kgdboc= on the same tty (but it still
>    works, just might cause troubles if you hit TX interrupt);

One of the nice features in FIQ debugger is the "console" command,
which causes all incoming serial characters to get passed to a console
device provided by the FIQ debugger, and characters from the console
to go out the serial port (when it is enabled).  That way you can
still have the console when you want it, but only one driver talking
to the hardware.

> 4. Address any comments. :-)
>
> Thanks!
>

[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[Anton Vorontsov]

On Thu, Jul 05, 2012 at 05:02:12PM -0700, Colin Cross wrote:
[...]
> KGDB can obviously only be enabled on development
> devices, although perhaps a more limited KDB could be left enabled.

Um, I would argue about 'obviously'. :-) It doesn't require
CONFIG_DEBUG_INFO (-g) or something like this, so if the concern is
the size (which is about 30..40 KB), then it is all manageable. If we
want it to be smaller, we should just work on making KGDB/KDB more
modular, so that we can exclude non-production features.

> > The FIQ debugger is a facility that can be used to debug situations
> > when the kernel stuck in uninterruptable sections, e.g. the kernel
> > infinitely loops or deadlocked in an interrupt or with interrupts
> > disabled. On some development boards there is even a special NMI
> > button, which is very useful for debugging weird kernel hangs.
> >
> > And FIQ is basically an NMI, it has a higher priority than IRQs, and
> > upon IRQ exception FIQs are not disabled. It is still possible to
> > disable FIQs (as well as some "NMIs" on other architectures), but via
> > special means.
> >
> > So, here FIQs and NMIs are synonyms, but in the code I use NMI term
> > for arch-independent code, and FIQs for ARM code.
> 
> Unfortunately, FIQs have been repurposed as secure interrupts on every
> ARM SoC that supports TrustZone, which is basically all of the latest
> generation, as well as a few of the previous generation.  When an FIQ
> arrives, the cpu traps into the secure mode, generally running a
> separate secure OS written by a 3rd party vendor.  We've tried to get
> some SoC secure implementations to drop out of secure mode and into
> the FIQ exception vector for specific irqs, with the registers set up
> to allow the FIQ handler to return back to the original execution
> point, but it's been successful.

It's pity, and this just shows that recent SOCs have a somewhat limited
debugging facilities. There are countless times when I was able to debug
a hang simply by hitting an NMI button on a reference board (not ARM,
tho) and just reading the trace.

Having any (even a watchdog) NMI connected to a kernel would be enough
to make things better. Btw, these patches' approach works even if we
can't reroute arbitrary interrupts to FIQs (like we do for serial lines).

[...]
> >   This might look as a drastic change, but it is not. There is actually
> >   no difference whether you have sync or async shell, or at least I
> >   couldn't find any use-case where this would matter at all. Anyways,
> >   it is still possible to do async shell in KDB, just don't see any
> >   need for this.
> 
> I think it could be an issue if KDB stopped execution whenever it
> received any character.  Serial ports are often noisy, especially when
> muxed over another port (we often use serial over the headset
> connector).  Noise on the async command line just causes characters
> that are ignored, on a command line that blocked execution noise would
> be catastrophic.

Aha, that's the real use-case, thanks! I started hacking the KDB
to add the async shell support, but then I realized that we still
don't need all the complexity. If the only purpose is to be safe from
the noise, then we can just do "knocking" before entering the debugger.

The thing is, we even have a standard sequence for entering KDB,
it is GDB-protocol command $3#33, so it actually makes sense to
implement this. This would be the only async command, and it doesn't
affect anything but the new code. I prepared a separate patch for this.

[...]
> One of the nice features in FIQ debugger is the "console" command,
> which causes all incoming serial characters to get passed to a console
> device provided by the FIQ debugger, and characters from the console
> to go out the serial port (when it is enabled).  That way you can
> still have the console when you want it, but only one driver talking
> to the hardware.

This is surely a nice feature, and we have a chance to make it work
with this scheme too. For example, we can reroute FIQ to IRQ, and
reinitialize the tty device, so that it would grab the IRQ, then we
can give uart back. Surely there are some implementation details
that makes it not that easy, but it is definitely doable.

Thanks!

-- 
Anton Vorontsov
Email: cbouatmailru at gmail.com

[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[Colin Cross]

On Fri, Jul 13, 2012 at 2:49 AM, Anton Vorontsov
<anton.vorontsov@linaro.org> wrote:
> On Thu, Jul 05, 2012 at 05:02:12PM -0700, Colin Cross wrote:
> [...]
>> KGDB can obviously only be enabled on development
>> devices, although perhaps a more limited KDB could be left enabled.
>
> Um, I would argue about 'obviously'. :-) It doesn't require
> CONFIG_DEBUG_INFO (-g) or something like this, so if the concern is
> the size (which is about 30..40 KB), then it is all manageable. If we
> want it to be smaller, we should just work on making KGDB/KDB more
> modular, so that we can exclude non-production features.

I was referring to the security implications, not size.  Leaving KDB
on is effectively instant root access over the serial console.

<snip

>> >   This might look as a drastic change, but it is not. There is actually
>> >   no difference whether you have sync or async shell, or at least I
>> >   couldn't find any use-case where this would matter at all. Anyways,
>> >   it is still possible to do async shell in KDB, just don't see any
>> >   need for this.
>>
>> I think it could be an issue if KDB stopped execution whenever it
>> received any character.  Serial ports are often noisy, especially when
>> muxed over another port (we often use serial over the headset
>> connector).  Noise on the async command line just causes characters
>> that are ignored, on a command line that blocked execution noise would
>> be catastrophic.
>
> Aha, that's the real use-case, thanks! I started hacking the KDB
> to add the async shell support, but then I realized that we still
> don't need all the complexity. If the only purpose is to be safe from
> the noise, then we can just do "knocking" before entering the debugger.
>
> The thing is, we even have a standard sequence for entering KDB,
> it is GDB-protocol command $3#33, so it actually makes sense to
> implement this. This would be the only async command, and it doesn't
> affect anything but the new code. I prepared a separate patch for this.

I would suggest making the sequence longer than just return.  A single
character is not that unlikely to be generated by random noise - I've
seen multiple devices reboot when the serial console was connected
because it received a SysRq-Crash (a break is all zeroes, which is
very common while shorting the lines as the console is plugged in, and
then random noise sent a 'c').

[PATCH 0/6] KGDB/KDB FIQ (NMI) debugger

[Anton Vorontsov]

On Fri, Jul 13, 2012 at 09:43:45AM -0700, Colin Cross wrote:
[...]
> I was referring to the security implications, not size.  Leaving KDB
> on is effectively instant root access over the serial console.

Oh, I see. Yes, for this we'd need to disable all modification commands.

[...]
> > The thing is, we even have a standard sequence for entering KDB,
> > it is GDB-protocol command $3#33, so it actually makes sense to
> > implement this. This would be the only async command, and it doesn't
> > affect anything but the new code. I prepared a separate patch for this.
> 
> I would suggest making the sequence longer than just return.  A single
> character is not that unlikely to be generated by random noise - I've
> seen multiple devices reboot when the serial console was connected
> because it received a SysRq-Crash (a break is all zeroes, which is
> very common while shorting the lines as the console is plugged in, and
> then random noise sent a 'c').

No no, it's not just return. It is either return or the longer '$3#33'
escape sequence. Default is $3#33, so it should be pretty safe (but of
course we can make it even longer, or even configurable).

Thanks,

-- 
Anton Vorontsov
Email: cbouatmailru at gmail.com