* Crash on armv7-a using KASAN
@ 2024-10-14 13:19 Clement LE GOFFIC
2024-10-15 7:55 ` Linus Walleij
2024-10-15 10:28 ` Mark Rutland
0 siblings, 2 replies; 21+ messages in thread
From: Clement LE GOFFIC @ 2024-10-14 13:19 UTC (permalink / raw)
To: Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Linus Walleij, Mark Brown,
linux-arm-kernel, linux-kernel
Cc: Ard Biesheuvel, linux-stm32, Antonio Borneo
Hello,
I have detected a kernel crash in latest kernel on armv7-a when Kasan is
enabled.
Git bisect points to Ard commit from 2021 :
b6506981f880 ARM: unwind: support unwinding across multiple stacks
ardb@kernel.org
This part of code is outside my expertise and I'm unable to propose a fix.
Below there is a simple way to replicate the crash inside qemu.
The offending commit is a preliminary patch of a series aimed at
managing multiple stacks.
The crash log from the offending commit differs with the crash log from
latest kernel due to the extra patches in the mentioned series.
Step to reproduce :
1. Ensure `qemu-system-arm` and `arm-none-eabi-gcc` are installed on
your system
2. Get a filesystem. I found a Debian installer filesystem (publicly
available):
$> wget
http://ftp.us.debian.org/debian/dists/stable/main/installer-armhf/current/images/cdrom/initrd.gz
3. Configure and build the kernel with KASAN enabled:
$> echo 'CONFIG_KASAN=y' > arch/arm/configs/fragment-kasan.config
$> make ARCH=arm CROSS_COMPILE=arm-none-eabi- vexpress_defconfig
fragment-kasan.config
$> make ARCH=arm CROSS_COMPILE=arm-none-eabi- -j16 all
4. Run the kernel in QEMU:
$> qemu-system-arm -kernel arch/arm/boot/zImage -machine vexpress-a15
-dtb arch/arm/boot/dts/arm/vexpress-v2p-ca15-tc1.dtb -append
"console=ttyAMA0" -initrd initrd.gz -m 512 -smp 2
5. When the "Select a language" menu appears, press ESC, in the new menu
select "Execute a shell", then "Continue"
6. The crash may appear directly during boot. If not, you can stress the
system with syscalls by running ten or more executions in parallel of:
$> find /usr/ -type f -exec grep randomStringToSearch {} \;&
7. Wait for a kernel crash.
Thank you for looking into this issue. Please let me know if you need
any further information.
Best regards,
Clément Le Goffic
NB: Two kernel crash dumps are following each other.
Crash log with recent kernel (v6.12-rc3) :
~ # Insufficient stack space to handle exception!
Task stack: [0xac000000..0xac004000]
IRQ stack: [0xa0808000..0xa080c000]
Overflow stack: [0x82964000..0x82965000]
Internal error: kernel stack overflow: 0 [#1] SMP ARM
Modules linked in:
CPU: 1 UID: 0 PID: 1678 Comm: grep Not tainted 6.12.0-rc3 #2
Hardware name: ARM-Versatile Express
PC is at do_translation_fault+0x30/0x2b0
LR is at do_DataAbort+0x74/0x1dc
pc : [<80125920>] lr : [<80125c14>] psr: 200f0193
sp : ac000008 ip : 00000051 fp : ac003d54
r10: ac003d20 r9 : 82412640 r8 : 00000005
r7 : ac000068 r6 : 7480002b r5 : ac000068 r4 : 7480002b
r3 : 74800015 r2 : ac000068 r1 : 00000005 r0 : ac0000a8
Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c5387d Table: 88a8406a DAC: 00000051
Register r0 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r1 information: non-paged memory
Register r2 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r3 information: non-paged memory
Register r4 information: non-paged memory
Register r5 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r6 information: non-paged memory
Register r7 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r8 information: non-paged memory
Register r9 information: non-slab/vmalloc memory
Register r10 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r11 information: 4-page vmalloc region starting at 0xac000000
allocated at kernel_clone+0x14c/0x768
Register r12 information: non-paged memory
Process grep (pid: 1678, stack limit = 0x3bfd4185)
Stack: (0xac000008 to 0xac004000)
0000: 00000000 00000000 00000000 00000005 00000005
7480002b
0020: ac000068 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0040: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00009c
00000005
0060: 83850900 80100aec ac000158 00000005 ac000118 7480002b 74800041
ac000118
0080: 74800041 ac000118 00000005 82412640 ac003d20 ac003d54 00000051
ac0000b8
00a0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
00c0: 00000000 00000005 00000005 74800041 ac000118 82412690 82412640
ac003d20
00e0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0100: 200f0193 ffffffff ac00014c 00000005 83850900 80100aec ac000208
00000005
0120: ac0001c8 74800041 74800057 ac0001c8 74800057 ac0001c8 00000005
82412640
0140: ac003d20 ac003d54 00000051 ac000168 80125c14 80125920 200f0193
ffffffff
0160: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800057
0180: ac0001c8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
01a0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0001fc
00000005
01c0: 83850900 80100aec ac0002b8 00000005 ac000278 74800057 7480006d
ac000278
01e0: 7480006d ac000278 00000005 82412640 ac003d20 ac003d54 00000051
ac000218
0200: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0220: 00000000 00000005 00000005 7480006d ac000278 82412690 82412640
ac003d20
0240: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0260: 200f0193 ffffffff ac0002ac 00000005 83850900 80100aec ac000368
00000005
0280: ac000328 7480006d 74800083 ac000328 74800083 ac000328 00000005
82412640
02a0: ac003d20 ac003d54 00000051 ac0002c8 80125c14 80125920 200f0193
ffffffff
02c0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800083
02e0: ac000328 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0300: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00035c
00000005
0320: 83850900 80100aec ac000418 00000005 ac0003d8 74800083 74800099
ac0003d8
0340: 74800099 ac0003d8 00000005 82412640 ac003d20 ac003d54 00000051
ac000378
0360: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0380: 00000000 00000005 00000005 74800099 ac0003d8 82412690 82412640
ac003d20
03a0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
03c0: 200f0193 ffffffff ac00040c 00000005 83850900 80100aec ac0004c8
00000005
03e0: ac000488 74800099 748000af ac000488 748000af ac000488 00000005
82412640
0400: ac003d20 ac003d54 00000051 ac000428 80125c14 80125920 200f0193
ffffffff
0420: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748000af
0440: ac000488 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0460: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0004bc
00000005
0480: 83850900 80100aec ac000578 00000005 ac000538 748000af 748000c5
ac000538
04a0: 748000c5 ac000538 00000005 82412640 ac003d20 ac003d54 00000051
ac0004d8
04c0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
04e0: 00000000 00000005 00000005 748000c5 ac000538 82412690 82412640
ac003d20
0500: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0520: 200f0193 ffffffff ac00056c 00000005 83850900 80100aec ac000628
00000005
0540: ac0005e8 748000c5 748000db ac0005e8 748000db ac0005e8 00000005
82412640
0560: ac003d20 ac003d54 00000051 ac000588 80125c14 80125920 200f0193
ffffffff
0580: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748000db
05a0: ac0005e8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
05c0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00061c
00000005
05e0: 83850900 80100aec ac0006d8 00000005 ac000698 748000db 748000f1
ac000698
0600: 748000f1 ac000698 00000005 82412640 ac003d20 ac003d54 00000051
ac000638
0620: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0640: 00000000 00000005 00000005 748000f1 ac000698 82412690 82412640
ac003d20
0660: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0680: 200f0193 ffffffff ac0006cc 00000005 83850900 80100aec ac000788
00000005
06a0: ac000748 748000f1 74800107 ac000748 74800107 ac000748 00000005
82412640
06c0: ac003d20 ac003d54 00000051 ac0006e8 80125c14 80125920 200f0193
ffffffff
06e0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800107
0700: ac000748 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0720: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00077c
00000005
0740: 83850900 80100aec ac000838 00000005 ac0007f8 74800107 7480011d
ac0007f8
0760: 7480011d ac0007f8 00000005 82412640 ac003d20 ac003d54 00000051
ac000798
0780: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
07a0: 00000000 00000005 00000005 7480011d ac0007f8 82412690 82412640
ac003d20
07c0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
07e0: 200f0193 ffffffff ac00082c 00000005 83850900 80100aec ac0008e8
00000005
0800: ac0008a8 7480011d 74800133 ac0008a8 74800133 ac0008a8 00000005
82412640
0820: ac003d20 ac003d54 00000051 ac000848 80125c14 80125920 200f0193
ffffffff
0840: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800133
0860: ac0008a8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0880: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0008dc
00000005
08a0: 83850900 80100aec ac000998 00000005 ac000958 74800133 74800149
ac000958
08c0: 74800149 ac000958 00000005 82412640 ac003d20 ac003d54 00000051
ac0008f8
08e0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0900: 00000000 00000005 00000005 74800149 ac000958 82412690 82412640
ac003d20
0920: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0940: 200f0193 ffffffff ac00098c 00000005 83850900 80100aec ac000a48
00000005
0960: ac000a08 74800149 7480015f ac000a08 7480015f ac000a08 00000005
82412640
0980: ac003d20 ac003d54 00000051 ac0009a8 80125c14 80125920 200f0193
ffffffff
09a0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480015f
09c0: ac000a08 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
09e0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac000a3c
00000005
0a00: 83850900 80100aec ac000af8 00000005 ac000ab8 7480015f 74800175
ac000ab8
0a20: 74800175 ac000ab8 00000005 82412640 ac003d20 ac003d54 00000051
ac000a58
0a40: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0a60: 00000000 00000005 00000005 74800175 ac000ab8 82412690 82412640
ac003d20
0a80: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0aa0: 200f0193 ffffffff ac000aec 00000005 83850900 80100aec ac000ba8
00000005
0ac0: ac000b68 74800175 7480018b ac000b68 7480018b ac000b68 00000005
82412640
0ae0: ac003d20 ac003d54 00000051 ac000b08 80125c14 80125920 200f0193
ffffffff
0b00: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480018b
0b20: ac000b68 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0b40: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac000b9c
00000005
0b60: 83850900 80100aec ac000c58 00000005 ac000c18 7480018b 748001a1
ac000c18
0b80: 748001a1 ac000c18 00000005 82412640 ac003d20 ac003d54 00000051
ac000bb8
0ba0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0bc0: 00000000 00000005 00000005 748001a1 ac000c18 82412690 82412640
ac003d20
0be0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0c00: 200f0193 ffffffff ac000c4c 00000005 83850900 80100aec ac000d08
00000005
0c20: ac000cc8 748001a1 748001b7 ac000cc8 748001b7 ac000cc8 00000005
82412640
0c40: ac003d20 ac003d54 00000051 ac000c68 80125c14 80125920 200f0193
ffffffff
0c60: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748001b7
0c80: ac000cc8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0ca0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac000cfc
00000005
0cc0: 83850900 80100aec ac000db8 00000005 ac000d78 748001b7 748001cd
ac000d78
0ce0: 748001cd ac000d78 00000005 82412640 ac003d20 ac003d54 00000051
ac000d18
0d00: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0d20: 00000000 00000005 00000005 748001cd ac000d78 82412690 82412640
ac003d20
0d40: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0d60: 200f0193 ffffffff ac000dac 00000005 83850900 80100aec ac000e68
00000005
0d80: ac000e28 748001cd 748001e3 ac000e28 748001e3 ac000e28 00000005
82412640
0da0: ac003d20 ac003d54 00000051 ac000dc8 80125c14 80125920 200f0193
ffffffff
0dc0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748001e3
0de0: ac000e28 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0e00: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac000e5c
00000005
0e20: 83850900 80100aec ac000f18 00000005 ac000ed8 748001e3 748001f9
ac000ed8
0e40: 748001f9 ac000ed8 00000005 82412640 ac003d20 ac003d54 00000051
ac000e78
0e60: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0e80: 00000000 00000005 00000005 748001f9 ac000ed8 82412690 82412640
ac003d20
0ea0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
0ec0: 200f0193 ffffffff ac000f0c 00000005 83850900 80100aec ac000fc8
00000005
0ee0: ac000f88 748001f9 7480020f ac000f88 7480020f ac000f88 00000005
82412640
0f00: ac003d20 ac003d54 00000051 ac000f28 80125c14 80125920 200f0193
ffffffff
0f20: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480020f
0f40: ac000f88 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
0f60: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac000fbc
00000005
0f80: 83850900 80100aec ac001078 00000005 ac001038 7480020f 74800225
ac001038
0fa0: 74800225 ac001038 00000005 82412640 ac003d20 ac003d54 00000051
ac000fd8
0fc0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
0fe0: 00000000 00000005 00000005 74800225 ac001038 82412690 82412640
ac003d20
1000: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1020: 200f0193 ffffffff ac00106c 00000005 83850900 80100aec ac001128
00000005
1040: ac0010e8 74800225 7480023b ac0010e8 7480023b ac0010e8 00000005
82412640
1060: ac003d20 ac003d54 00000051 ac001088 80125c14 80125920 200f0193
ffffffff
1080: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480023b
10a0: ac0010e8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
10c0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00111c
00000005
10e0: 83850900 80100aec ac0011d8 00000005 ac001198 7480023b 74800251
ac001198
1100: 74800251 ac001198 00000005 82412640 ac003d20 ac003d54 00000051
ac001138
1120: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1140: 00000000 00000005 00000005 74800251 ac001198 82412690 82412640
ac003d20
1160: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1180: 200f0193 ffffffff ac0011cc 00000005 83850900 80100aec ac001288
00000005
11a0: ac001248 74800251 74800267 ac001248 74800267 ac001248 00000005
82412640
11c0: ac003d20 ac003d54 00000051 ac0011e8 80125c14 80125920 200f0193
ffffffff
11e0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800267
1200: ac001248 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1220: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00127c
00000005
1240: 83850900 80100aec ac001338 00000005 ac0012f8 74800267 7480027d
ac0012f8
1260: 7480027d ac0012f8 00000005 82412640 ac003d20 ac003d54 00000051
ac001298
1280: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
12a0: 00000000 00000005 00000005 7480027d ac0012f8 82412690 82412640
ac003d20
12c0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
12e0: 200f0193 ffffffff ac00132c 00000005 83850900 80100aec ac0013e8
00000005
1300: ac0013a8 7480027d 74800293 ac0013a8 74800293 ac0013a8 00000005
82412640
1320: ac003d20 ac003d54 00000051 ac001348 80125c14 80125920 200f0193
ffffffff
1340: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800293
1360: ac0013a8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1380: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0013dc
00000005
13a0: 83850900 80100aec ac001498 00000005 ac001458 74800293 748002a9
ac001458
13c0: 748002a9 ac001458 00000005 82412640 ac003d20 ac003d54 00000051
ac0013f8
13e0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1400: 00000000 00000005 00000005 748002a9 ac001458 82412690 82412640
ac003d20
1420: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1440: 200f0193 ffffffff ac00148c 00000005 83850900 80100aec ac001548
00000005
1460: ac001508 748002a9 748002bf ac001508 748002bf ac001508 00000005
82412640
1480: ac003d20 ac003d54 00000051 ac0014a8 80125c14 80125920 200f0193
ffffffff
14a0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748002bf
14c0: ac001508 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
14e0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00153c
00000005
1500: 83850900 80100aec ac0015f8 00000005 ac0015b8 748002bf 748002d5
ac0015b8
1520: 748002d5 ac0015b8 00000005 82412640 ac003d20 ac003d54 00000051
ac001558
1540: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1560: 00000000 00000005 00000005 748002d5 ac0015b8 82412690 82412640
ac003d20
1580: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
15a0: 200f0193 ffffffff ac0015ec 00000005 83850900 80100aec ac0016a8
00000005
15c0: ac001668 748002d5 748002eb ac001668 748002eb ac001668 00000005
82412640
15e0: ac003d20 ac003d54 00000051 ac001608 80125c14 80125920 200f0193
ffffffff
1600: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748002eb
1620: ac001668 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1640: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00169c
00000005
1660: 83850900 80100aec ac001758 00000005 ac001718 748002eb 74800301
ac001718
1680: 74800301 ac001718 00000005 82412640 ac003d20 ac003d54 00000051
ac0016b8
16a0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
16c0: 00000000 00000005 00000005 74800301 ac001718 82412690 82412640
ac003d20
16e0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1700: 200f0193 ffffffff ac00174c 00000005 83850900 80100aec ac001808
00000005
1720: ac0017c8 74800301 74800317 ac0017c8 74800317 ac0017c8 00000005
82412640
1740: ac003d20 ac003d54 00000051 ac001768 80125c14 80125920 200f0193
ffffffff
1760: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800317
1780: ac0017c8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
17a0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0017fc
00000005
17c0: 83850900 80100aec ac0018b8 00000005 ac001878 74800317 7480032d
ac001878
17e0: 7480032d ac001878 00000005 82412640 ac003d20 ac003d54 00000051
ac001818
1800: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1820: 00000000 00000005 00000005 7480032d ac001878 82412690 82412640
ac003d20
1840: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1860: 200f0193 ffffffff ac0018ac 00000005 83850900 80100aec ac001968
00000005
1880: ac001928 7480032d 74800343 ac001928 74800343 ac001928 00000005
82412640
18a0: ac003d20 ac003d54 00000051 ac0018c8 80125c14 80125920 200f0193
ffffffff
18c0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800343
18e0: ac001928 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1900: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00195c
00000005
1920: 83850900 80100aec ac001a18 00000005 ac0019d8 74800343 74800359
ac0019d8
1940: 74800359 ac0019d8 00000005 82412640 ac003d20 ac003d54 00000051
ac001978
1960: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1980: 00000000 00000005 00000005 74800359 ac0019d8 82412690 82412640
ac003d20
19a0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
19c0: 200f0193 ffffffff ac001a0c 00000005 83850900 80100aec ac001ac8
00000005
19e0: ac001a88 74800359 7480036f ac001a88 7480036f ac001a88 00000005
82412640
1a00: ac003d20 ac003d54 00000051 ac001a28 80125c14 80125920 200f0193
ffffffff
1a20: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480036f
1a40: ac001a88 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1a60: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac001abc
00000005
1a80: 83850900 80100aec ac001b78 00000005 ac001b38 7480036f 74800385
ac001b38
1aa0: 74800385 ac001b38 00000005 82412640 ac003d20 ac003d54 00000051
ac001ad8
1ac0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1ae0: 00000000 00000005 00000005 74800385 ac001b38 82412690 82412640
ac003d20
1b00: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1b20: 200f0193 ffffffff ac001b6c 00000005 83850900 80100aec ac001c28
00000005
1b40: ac001be8 74800385 7480039b ac001be8 7480039b ac001be8 00000005
82412640
1b60: ac003d20 ac003d54 00000051 ac001b88 80125c14 80125920 200f0193
ffffffff
1b80: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480039b
1ba0: ac001be8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1bc0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac001c1c
00000005
1be0: 83850900 80100aec ac001cd8 00000005 ac001c98 7480039b 748003b1
ac001c98
1c00: 748003b1 ac001c98 00000005 82412640 ac003d20 ac003d54 00000051
ac001c38
1c20: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1c40: 00000000 00000005 00000005 748003b1 ac001c98 82412690 82412640
ac003d20
1c60: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1c80: 200f0193 ffffffff ac001ccc 00000005 83850900 80100aec ac001d88
00000005
1ca0: ac001d48 748003b1 748003c7 ac001d48 748003c7 ac001d48 00000005
82412640
1cc0: ac003d20 ac003d54 00000051 ac001ce8 80125c14 80125920 200f0193
ffffffff
1ce0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748003c7
1d00: ac001d48 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1d20: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac001d7c
00000005
1d40: 83850900 80100aec ac001e38 00000005 ac001df8 748003c7 748003dd
ac001df8
1d60: 748003dd ac001df8 00000005 82412640 ac003d20 ac003d54 00000051
ac001d98
1d80: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1da0: 00000000 00000005 00000005 748003dd ac001df8 82412690 82412640
ac003d20
1dc0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1de0: 200f0193 ffffffff ac001e2c 00000005 83850900 80100aec ac001ee8
00000005
1e00: ac001ea8 748003dd 748003f3 ac001ea8 748003f3 ac001ea8 00000005
82412640
1e20: ac003d20 ac003d54 00000051 ac001e48 80125c14 80125920 200f0193
ffffffff
1e40: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748003f3
1e60: ac001ea8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1e80: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac001edc
00000005
1ea0: 83850900 80100aec ac001f98 00000005 ac001f58 748003f3 74800409
ac001f58
1ec0: 74800409 ac001f58 00000005 82412640 ac003d20 ac003d54 00000051
ac001ef8
1ee0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
1f00: 00000000 00000005 00000005 74800409 ac001f58 82412690 82412640
ac003d20
1f20: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
1f40: 200f0193 ffffffff ac001f8c 00000005 83850900 80100aec ac002048
00000005
1f60: ac002008 74800409 7480041f ac002008 7480041f ac002008 00000005
82412640
1f80: ac003d20 ac003d54 00000051 ac001fa8 80125c14 80125920 200f0193
ffffffff
1fa0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480041f
1fc0: ac002008 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
1fe0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00203c
00000005
2000: 83850900 80100aec ac0020f8 00000005 ac0020b8 7480041f 74800435
ac0020b8
2020: 74800435 ac0020b8 00000005 82412640 ac003d20 ac003d54 00000051
ac002058
2040: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2060: 00000000 00000005 00000005 74800435 ac0020b8 82412690 82412640
ac003d20
2080: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
20a0: 200f0193 ffffffff ac0020ec 00000005 83850900 80100aec ac0021a8
00000005
20c0: ac002168 74800435 7480044b ac002168 7480044b ac002168 00000005
82412640
20e0: ac003d20 ac003d54 00000051 ac002108 80125c14 80125920 200f0193
ffffffff
2100: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480044b
2120: ac002168 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2140: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00219c
00000005
2160: 83850900 80100aec ac002258 00000005 ac002218 7480044b 74800461
ac002218
2180: 74800461 ac002218 00000005 82412640 ac003d20 ac003d54 00000051
ac0021b8
21a0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
21c0: 00000000 00000005 00000005 74800461 ac002218 82412690 82412640
ac003d20
21e0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2200: 200f0193 ffffffff ac00224c 00000005 83850900 80100aec ac002308
00000005
2220: ac0022c8 74800461 74800477 ac0022c8 74800477 ac0022c8 00000005
82412640
2240: ac003d20 ac003d54 00000051 ac002268 80125c14 80125920 200f0193
ffffffff
2260: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800477
2280: ac0022c8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
22a0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0022fc
00000005
22c0: 83850900 80100aec ac0023b8 00000005 ac002378 74800477 7480048d
ac002378
22e0: 7480048d ac002378 00000005 82412640 ac003d20 ac003d54 00000051
ac002318
2300: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2320: 00000000 00000005 00000005 7480048d ac002378 82412690 82412640
ac003d20
2340: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2360: 200f0193 ffffffff ac0023ac 00000005 83850900 80100aec ac002468
00000005
2380: ac002428 7480048d 748004a3 ac002428 748004a3 ac002428 00000005
82412640
23a0: ac003d20 ac003d54 00000051 ac0023c8 80125c14 80125920 200f0193
ffffffff
23c0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748004a3
23e0: ac002428 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2400: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00245c
00000005
2420: 83850900 80100aec ac002518 00000005 ac0024d8 748004a3 748004b9
ac0024d8
2440: 748004b9 ac0024d8 00000005 82412640 ac003d20 ac003d54 00000051
ac002478
2460: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2480: 00000000 00000005 00000005 748004b9 ac0024d8 82412690 82412640
ac003d20
24a0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
24c0: 200f0193 ffffffff ac00250c 00000005 83850900 80100aec ac0025c8
00000005
24e0: ac002588 748004b9 748004cf ac002588 748004cf ac002588 00000005
82412640
2500: ac003d20 ac003d54 00000051 ac002528 80125c14 80125920 200f0193
ffffffff
2520: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748004cf
2540: ac002588 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2560: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0025bc
00000005
2580: 83850900 80100aec ac002678 00000005 ac002638 748004cf 748004e5
ac002638
25a0: 748004e5 ac002638 00000005 82412640 ac003d20 ac003d54 00000051
ac0025d8
25c0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
25e0: 00000000 00000005 00000005 748004e5 ac002638 82412690 82412640
ac003d20
2600: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2620: 200f0193 ffffffff ac00266c 00000005 83850900 80100aec ac002728
00000005
2640: ac0026e8 748004e5 748004fb ac0026e8 748004fb ac0026e8 00000005
82412640
2660: ac003d20 ac003d54 00000051 ac002688 80125c14 80125920 200f0193
ffffffff
2680: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748004fb
26a0: ac0026e8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
26c0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00271c
00000005
26e0: 83850900 80100aec ac0027d8 00000005 ac002798 748004fb 74800511
ac002798
2700: 74800511 ac002798 00000005 82412640 ac003d20 ac003d54 00000051
ac002738
2720: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2740: 00000000 00000005 00000005 74800511 ac002798 82412690 82412640
ac003d20
2760: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2780: 200f0193 ffffffff ac0027cc 00000005 83850900 80100aec ac002888
00000005
27a0: ac002848 74800511 74800527 ac002848 74800527 ac002848 00000005
82412640
27c0: ac003d20 ac003d54 00000051 ac0027e8 80125c14 80125920 200f0193
ffffffff
27e0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800527
2800: ac002848 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2820: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac00287c
00000005
2840: 83850900 80100aec ac002938 00000005 ac0028f8 74800527 7480053d
ac0028f8
2860: 7480053d ac0028f8 00000005 82412640 ac003d20 ac003d54 00000051
ac002898
2880: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
28a0: 00000000 00000005 00000005 7480053d ac0028f8 82412690 82412640
ac003d20
28c0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
28e0: 200f0193 ffffffff ac00292c 00000005 83850900 80100aec ac0029e8
00000005
2900: ac0029a8 7480053d 74800553 ac0029a8 74800553 ac0029a8 00000005
82412640
2920: ac003d20 ac003d54 00000051 ac002948 80125c14 80125920 200f0193
ffffffff
2940: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800553
2960: ac0029a8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2980: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac0029dc
00000005
29a0: 83850900 80100aec ac002a98 00000005 ac002a58 74800553 74800569
ac002a58
29c0: 74800569 ac002a58 00000005 82412640 ac003d20 ac003d54 00000051
ac0029f8
29e0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2a00: 00000000 00000005 00000005 74800569 ac002a58 82412690 82412640
ac003d20
2a20: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2a40: 200f0193 ffffffff ac002a8c 00000005 83850900 80100aec ac002b48
00000005
2a60: ac002b08 74800569 7480057f ac002b08 7480057f ac002b08 00000005
82412640
2a80: ac003d20 ac003d54 00000051 ac002aa8 80125c14 80125920 200f0193
ffffffff
2aa0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480057f
2ac0: ac002b08 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2ae0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac002b3c
00000005
2b00: 83850900 80100aec ac002bf8 00000005 ac002bb8 7480057f 74800595
ac002bb8
2b20: 74800595 ac002bb8 00000005 82412640 ac003d20 ac003d54 00000051
ac002b58
2b40: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2b60: 00000000 00000005 00000005 74800595 ac002bb8 82412690 82412640
ac003d20
2b80: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2ba0: 200f0193 ffffffff ac002bec 00000005 83850900 80100aec ac002ca8
00000005
2bc0: ac002c68 74800595 748005ab ac002c68 748005ab ac002c68 00000005
82412640
2be0: ac003d20 ac003d54 00000051 ac002c08 80125c14 80125920 200f0193
ffffffff
2c00: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748005ab
2c20: ac002c68 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2c40: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac002c9c
00000005
2c60: 83850900 80100aec ac002d58 00000005 ac002d18 748005ab 748005c1
ac002d18
2c80: 748005c1 ac002d18 00000005 82412640 ac003d20 ac003d54 00000051
ac002cb8
2ca0: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2cc0: 00000000 00000005 00000005 748005c1 ac002d18 82412690 82412640
ac003d20
2ce0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2d00: 200f0193 ffffffff ac002d4c 00000005 83850900 80100aec ac002e08
00000005
2d20: ac002dc8 748005c1 748005d7 ac002dc8 748005d7 ac002dc8 00000005
82412640
2d40: ac003d20 ac003d54 00000051 ac002d68 80125c14 80125920 200f0193
ffffffff
2d60: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
748005d7
2d80: ac002dc8 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2da0: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac002dfc
00000005
2dc0: 83850900 80100aec ac002eb8 00000005 ac002e78 748005d7 748005ed
ac002e78
2de0: 748005ed ac002e78 00000005 82412640 ac003d20 ac003d54 00000051
ac002e18
2e00: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2e20: 00000000 00000005 00000005 748005ed ac002e78 82412690 82412640
ac003d20
2e40: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2e60: 200f0193 ffffffff ac002eac 00000005 83850900 80100aec ac002f68
00000005
2e80: ac002f28 748005ed 74800603 ac002f28 74800603 ac002f28 00000005
82412640
2ea0: ac003d20 ac003d54 00000051 ac002ec8 80125c14 80125920 200f0193
ffffffff
2ec0: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
74800603
2ee0: ac002f28 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
00000000
2f00: 00000000 00000000 00000000 80125920 200f0193 ffffffff ac002f5c
00000005
2f20: 83850900 80100aec ac003018 00000005 ac002fd8 74800603 74800619
ac002fd8
2f40: 74800619 ac002fd8 00000005 82412640 ac003d20 ac003d54 00000051
ac002f78
2f60: 80125c14 80125920 200f0193 ffffffff 00000051 00000000 00000000
00000000
2f80: 00000000 00000005 00000005 74800619 ac002fd8 82412690 82412640
ac003d20
2fa0: ac003d54 80125c14 00000000 00000000 00000000 00000000 00000000
80125920
2fc0: 200f0193 ffffffff ac00300c 00000005 83850900 80100aec ac0030c8
00000005
2fe0: ac003088 74800619 7480062f ac003088 7480062f ac003088 00000005
82412640
3000: ac003d20 ac003d54 00000051 ac003028 80125c14 80125920 200f0193
ffffffff
3020: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480062f
3040: ac003088 82412690 82412640 ac003d20 ac003d54 80125c14 00000200
8011e3fc
3060: 00000000 ac003180 ac0031c0 80125920 200f0193 ffffffff ac0030bc
00000005
3080: 83850900 80100aec ac003178 00000005 ac003138 7480062f 74800645
ac003138
30a0: 74800645 ac003138 00000005 82412640 ac003d20 ac003d54 00000051
ac0030d8
30c0: 80125c14 80125920 200f0193 ffffffff 00000051 ac0031f8 ac003120
82249adc
30e0: 00000001 00000005 00000005 74800645 ac003138 82412690 82412640
ac003d20
3100: ac003d54 80125c14 8011e2b0 0000001e 00000000 ac003fa8 80100060
80125920
3120: 200f0193 ffffffff ac00316c 00000005 83850900 80100aec ac003228
00000005
3140: ac0031e8 74800645 7480065b ac0031e8 7480065b ac0031e8 00000005
82412640
3160: ac003d20 ac003d54 00000051 ac003188 80125c14 80125920 200f0193
ffffffff
3180: 00000051 00000000 00000000 00000000 00000000 00000005 00000005
7480065b
31a0: ac0031e8 82412690 82412640 ac003d20 ac003d54 80125c14 ac0031ec
801151c0
31c0: 0000001e 7bdcba80 80100060 80125920 200f0193 ffffffff ac00321c
00000005
31e0: 83850900 80100aec ac0032d8 00000005 ac003298 7480065b 74800671
ac003298
3200: 74800671 ac003298 00000005 82412640 ac003d20 ac003d54 00000051
ac003238
3220: 80125c14 80125920 200f0193 ffffffff 00000051 ac003358 ac003280
82249adc
3240: 41b58ab3 00000005 00000005 74800671 ac003298 82412690 82412640
ac003d20
3260: ac003d54 80125c14 82802700 85709204 00000000 80fc8c00 00000000
80125920
3280: 200f0193 ffffffff ac0032cc 00000005 83850900 80100aec ac003388
00000005
32a0: ac003348 74800671 74800687 ac003348 74800687 ac003348 00000005
82412640
32c0: ac003d20 ac003d54 00000051 ac0032e8 80125c14 80125920 200f0193
ffffffff
32e0: 00000051 8068afe4 80309890 8269af00 83850900 00000005 00000005
74800687
3300: ac003348 82412690 82412640 ac003d20 ac003d54 80125c14 7480066c
83850900
3320: 00000004 1580068c ac00346c 80125920 200f0193 ffffffff ac00337c
00000005
3340: 83850900 80100aec ac003438 00000005 ac0033f8 74800687 7480069d
ac0033f8
3360: 7480069d ac0033f8 00000005 82412640 ac003d20 ac003d54 00000051
ac003398
3380: 80125c14 80125920 200f0193 ffffffff 00000051 00000001 6ede14f8
0000000b
33a0: 80100290 00000005 00000005 7480069d ac0033f8 82412690 82412640
ac003d20
33c0: ac003d54 80125c14 0000000b ac003500 ac003528 ac003fa8 00000004
80125920
33e0: 200f0193 ffffffff ac00342c 00000005 83850900 80100aec ac0034e8
00000005
3400: ac0034a8 7480069d 748006b3 ac0034a8 748006b3 ac0034a8 00000005
82412640
3420: ac003d20 ac003d54 00000051 ac003448 80125c14 80125920 200f0193
ffffffff
3440: 00000051 8215f74c 8011e2b0 00000200 9ed04c40 00000005 00000005
748006b3
3460: ac0034a8 82412690 82412640 ac003d20 ac003d54 80125c14 00002000
000000c0
3480: 80100290 83850900 80309890 80125920 200f0193 ffffffff ac0034dc
00000005
34a0: 83850900 80100aec ac003598 00000005 ac003558 748006b3 748006c9
ac003558
34c0: 748006c9 ac003558 00000005 82412640 ac003d20 ac003d54 00000051
ac0034f8
34e0: 80125c14 80125920 200f0193 ffffffff 00000051 82276320 82802700
7bdcba80
3500: 41b58ab3 00000005 00000005 748006c9 ac003558 82412690 82412640
ac003d20
3520: ac003d54 80125c14 ac003680 ac003660 ac003734 80100bb8 806a51f4
80125920
3540: 200f0193 ffffffff ac00358c 00000005 83850900 80100aec ac003648
00000005
3560: ac003608 748006c9 748006df ac003608 748006df ac003608 00000005
82412640
3580: ac003d20 ac003d54 00000051 ac0035a8 80125c14 80125920 200f0193
ffffffff
35a0: 00000051 8011e3fc 8011e2b0 ac0036e0 ac003708 00000005 00000005
748006df
35c0: ac003608 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
158006e1
35e0: 00000000 158006e2 ac003fa4 80125920 200f0193 ffffffff ac00363c
00000005
3600: 83850900 80100aec ac0036f8 00000005 ac0036b8 748006df 748006f5
ac0036b8
3620: 748006f5 ac0036b8 00000005 82412640 ac003d20 ac003d54 00000051
ac003658
3640: 80125c14 80125920 200f0193 ffffffff 00000051 6eda3cf8 66daa3fc
0000018d
3660: 80100290 00000005 00000005 748006f5 ac0036b8 82412690 82412640
ac003d20
3680: ac003d54 80125c14 66dc9d80 ac003fa8 80100060 80100060 ac003fa4
80125920
36a0: 200f0193 ffffffff ac0036ec 00000005 83850900 80100aec ac0037a8
00000005
36c0: ac003768 748006f5 7480070b ac003768 7480070b ac003768 00000005
82412640
36e0: ac003d20 ac003d54 00000051 ac003708 80125c14 80125920 200f0193
ffffffff
3700: 00000051 801151c0 66dc9d80 ac003fa8 80100060 00000005 00000005
7480070b
3720: ac003768 82412690 82412640 ac003d20 ac003d54 80125c14 00000000
82fde708
3740: 41b58ab3 8216c648 80309b10 80125920 200f0193 ffffffff ac00379c
00000005
3760: 83850900 80100aec ac003858 00000005 ac003818 7480070b 74800721
ac003818
3780: 74800721 ac003818 00000005 82412640 ac003d20 ac003d54 00000051
ac0037b8
37a0: 80125c14 80125920 200f0193 ffffffff 00000051 828c3900 82fde784
000000c8
37c0: 9ec51bc0 00000005 00000005 74800721 ac003818 82412690 82412640
ac003d20
37e0: ac003d54 80125c14 806a8094 806a9af4 806aaf30 80684504 80684d98
80125920
3800: 200f0193 ffffffff ac00384c 00000005 83850900 80100aec ac003908
00000005
3820: ac0038c8 74800721 74800737 ac0038c8 74800737 ac0038c8 00000005
82412640
3840: ac003d20 ac003d54 00000051 ac003868 80125c14 80125920 200f0193
ffffffff
3860: 00000051 00000cc0 00000003 806c0b7c 894fa229 00000005 00000005
74800737
3880: ac0038c8 82412690 82412640 ac003d20 ac003d54 80125c14 9ab7c3b8
83850900
38a0: 74800728 802dc220 9ab7c340 80125920 200f0193 ffffffff ac0038fc
00000005
38c0: 83850900 80100aec ac0039b8 00000005 ac003978 74800737 7480074d
ac003978
38e0: 7480074d ac003978 00000005 82412640 ac003d20 ac003d54 00000051
ac003918
3900: 80125c14 80125920 200f0193 ffffffff 00000051 00000004 10488cc6
00000004
3920: 1356f86a 00000005 00000005 7480074d ac003978 82412690 82412640
ac003d20
3940: ac003d54 80125c14 802d3974 00000000 00000000 80dc7c84 00000000
80125920
3960: 200f0193 ffffffff ac0039ac 00000005 83850900 80100aec ac003a68
00000005
3980: ac003a28 7480074d 74800763 ac003a28 74800763 ac003a28 00000005
82412640
39a0: ac003d20 ac003d54 00000051 ac0039c8 80125c14 80125920 200f0193
ffffffff
39c0: 00000051 82e747d0 00000000 82fde768 6f5fbced 00000005 00000005
74800763
39e0: ac003a28 82412690 82412640 ac003d20 ac003d54 80125c14 74800750
828d0110
3a00: ac003c24 00000004 00000000 80125920 200f0193 ffffffff ac003a5c
00000005
3a20: 83850900 80100aec ac003b18 00000005 ac003ad8 74800763 74800779
ac003ad8
3a40: 74800779 ac003ad8 00000005 82412640 ac003d20 ac003d54 00000051
ac003a78
3a60: 80125c14 80125920 200f0193 ffffffff 00000051 74800754 ac003b00
ac003ac0
3a80: 006f006f 00000005 00000005 74800779 ac003ad8 82412690 82412640
ac003d20
3aa0: ac003d54 80125c14 80dc7bb8 ac003c24 00000000 15800784 82e747d0
80125920
3ac0: 200f0193 ffffffff ac003b0c 00000005 83850900 80100aec ac003bc8
00000005
3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
82412640
3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
ffffffff
3b20: 00000051 15800780 ac003c00 7bdcba80 ac003c24 00000005 00000005
7480078f
3b40: ac003b88 82412690 82412640 ac003d20 ac003d54 80125c14 15800788
00000004
3b60: ac003c40 9ed1bf1c 187c7000 80125920 200f0193 ffffffff ac003bbc
00000805
3b80: 83850900 80100aec ac003c78 00000805 ac003c38 7480078f 74800798
ac003c38
3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
ac003bd8
3bc0: 80125c14 80125920 200f0193 ffffffff 00000051 833bca80 74800788
80231e84
3be0: 00000001 00000005 00000805 74800798 ac003c38 82412690 82412640
ac003d20
3c00: ac003d54 80125c14 81e1c9c0 81e1cc60 ffffffe1 0000001f 00000001
801e7c40
3c20: 600f0093 ffffffff ac003c6c 9ab7bb00 83850900 80100aec 9ab7bb00
8652c000
3c40: f1f1f1f1 f3f3f304 833bc800 9ab7bb00 83850900 833bcc88 9ab7bb00
74800798
3c60: ac003d20 ac003d54 00000000 ac003c88 81dceb28 801e7c40 600f0093
ffffffff
3c80: 00000051 9ab7c020 1356f804 8023a128 6eda3b20 6eda3cf8 00000000
7bdcba80
3ca0: 83850b80 9ab7bb40 833bca80 8652c000 10677960 00000001 00000019
00000004
3cc0: 41b58ab3 82164e70 801e7be4 833bca88 8652c240 9ab730e0 0000098f
00000000
3ce0: 83850d8c 00000004 824071f8 8012f208 9ab7bb00 8652c000 00000001
823ac0e0
3d00: 00000004 00000001 00000001 80205ed0 802058bc 833bc800 9ab7bb00
821e6a04
3d20: 83850900 83850d24 1070a120 0000000a ac003e74 833bc800 9ab7bb00
83850900
3d40: 833bcc88 823ba890 823ba898 8652fb80 ac003e74 81dceb28 80100060
80100060
3d60: ac003fa4 7bdcba80 9ab7c0b0 83850908 9ab7c0ac 1356f816 187c7000
1070a1a4
3d80: 00000004 81dd0274 1070a121 748007b8 9ab7c178 1356f82f 8652fb80
8652c000
3da0: 824071f8 10677991 ac003e40 83850900 894f9100 894f9108 894fb200
00000cc0
3dc0: 41b58ab3 82164e84 81dcde74 826b33c0 00000001 80fc8c00 00000000
00000000
3de0: 00027e2a 9a302000 6eda3b20 80684cfc 00000000 894fa204 00000000
9ed1bf00
3e00: 81e11588 00000000 00000002 00000000 66dc9d80 8064abcc 8064abb8
8064c744
3e20: 8064adac 8062601c 80684fa8 80100060 00000001 00000000 13da4b88
748007cc
3e40: 83850900 6eda3fe4 00000000 7bdcba80 8941e600 83850900 00000003
6f70a120
3e60: 83850b08 80100290 ac003fb0 748007dc ac003e8c 81dd0274 1070a120
00000000
3e80: ac003ff0 158007fe 83850900 801143c8 83850900 ac003f80 748007dc
83850900
3ea0: 00000003 6edffd08 6edffc88 5ac3c35a 826b33dc 828c3800 400f0013
400f0013
3ec0: 41b58ab3 82180a68 80666bfc 806222f4 00000cc0 806aa27c ffffffff
00000cc0
3ee0: 41b58ab3 8215f008 8011430c 00000000 00000000 00000000 00000000
00000000
3f00: 894fa200 828c3800 6ee00000 00000ff0 00002200 00001100 7029f442
7029f440
3f20: 828c3800 894fa200 9ed1bf00 826b33c0 80684fa8 00000000 00000000
8062601c
3f40: 894fa200 894fa208 894fa200 894fa208 894fa200 000007ff 6eda3b20
fffffffe
3f60: ffffff9c 894fa200 000007ff 6eda3b20 83850900 0000018d 66dc9d80
80684fa8
3f80: 6eda3b20 7bdcba80 00000000 6eda3b20 6eda3cf8 66daa3fc 0000018d
80100290
3fa0: 83850900 0000018d 66dc9d80 80100088 fffffffe 6eda3c80 00000800
000007ff
3fc0: 6eda3b20 6eda3cf8 66daa3fc 0000018d ffffff9c 00000000 00000000
66dc9d80
3fe0: 0000018d 6eda3af8 66dc09cf 66dbff66 600f0030 ffffff9c 00000000
00000000
Call trace:
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000068 to 0xac0000b0)
0060: ac000158 00000005 ac000118 7480002b 74800041
ac000118
0080: 74800041 ac000118 00000005 82412640 ac003d20 ac003d54 00000051
ac0000b8
00a0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000118 to 0xac000160)
0100: ac000208
00000005
0120: ac0001c8 74800041 74800057 ac0001c8 74800057 ac0001c8 00000005
82412640
0140: ac003d20 ac003d54 00000051 ac000168 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0001c8 to 0xac000210)
01c0: ac0002b8 00000005 ac000278 74800057 7480006d
ac000278
01e0: 7480006d ac000278 00000005 82412640 ac003d20 ac003d54 00000051
ac000218
0200: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000278 to 0xac0002c0)
0260: ac000368
00000005
0280: ac000328 7480006d 74800083 ac000328 74800083 ac000328 00000005
82412640
02a0: ac003d20 ac003d54 00000051 ac0002c8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000328 to 0xac000370)
0320: ac000418 00000005 ac0003d8 74800083 74800099
ac0003d8
0340: 74800099 ac0003d8 00000005 82412640 ac003d20 ac003d54 00000051
ac000378
0360: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0003d8 to 0xac000420)
03c0: ac0004c8
00000005
03e0: ac000488 74800099 748000af ac000488 748000af ac000488 00000005
82412640
0400: ac003d20 ac003d54 00000051 ac000428 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000488 to 0xac0004d0)
0480: ac000578 00000005 ac000538 748000af 748000c5
ac000538
04a0: 748000c5 ac000538 00000005 82412640 ac003d20 ac003d54 00000051
ac0004d8
04c0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000538 to 0xac000580)
0520: ac000628
00000005
0540: ac0005e8 748000c5 748000db ac0005e8 748000db ac0005e8 00000005
82412640
0560: ac003d20 ac003d54 00000051 ac000588 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0005e8 to 0xac000630)
05e0: ac0006d8 00000005 ac000698 748000db 748000f1
ac000698
0600: 748000f1 ac000698 00000005 82412640 ac003d20 ac003d54 00000051
ac000638
0620: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000698 to 0xac0006e0)
0680: ac000788
00000005
06a0: ac000748 748000f1 74800107 ac000748 74800107 ac000748 00000005
82412640
06c0: ac003d20 ac003d54 00000051 ac0006e8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000748 to 0xac000790)
0740: ac000838 00000005 ac0007f8 74800107 7480011d
ac0007f8
0760: 7480011d ac0007f8 00000005 82412640 ac003d20 ac003d54 00000051
ac000798
0780: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0007f8 to 0xac000840)
07e0: ac0008e8
00000005
0800: ac0008a8 7480011d 74800133 ac0008a8 74800133 ac0008a8 00000005
82412640
0820: ac003d20 ac003d54 00000051 ac000848 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0008a8 to 0xac0008f0)
08a0: ac000998 00000005 ac000958 74800133 74800149
ac000958
08c0: 74800149 ac000958 00000005 82412640 ac003d20 ac003d54 00000051
ac0008f8
08e0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000958 to 0xac0009a0)
0940: ac000a48
00000005
0960: ac000a08 74800149 7480015f ac000a08 7480015f ac000a08 00000005
82412640
0980: ac003d20 ac003d54 00000051 ac0009a8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000a08 to 0xac000a50)
0a00: ac000af8 00000005 ac000ab8 7480015f 74800175
ac000ab8
0a20: 74800175 ac000ab8 00000005 82412640 ac003d20 ac003d54 00000051
ac000a58
0a40: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000ab8 to 0xac000b00)
0aa0: ac000ba8
00000005
0ac0: ac000b68 74800175 7480018b ac000b68 7480018b ac000b68 00000005
82412640
0ae0: ac003d20 ac003d54 00000051 ac000b08 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000b68 to 0xac000bb0)
0b60: ac000c58 00000005 ac000c18 7480018b 748001a1
ac000c18
0b80: 748001a1 ac000c18 00000005 82412640 ac003d20 ac003d54 00000051
ac000bb8
0ba0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000c18 to 0xac000c60)
0c00: ac000d08
00000005
0c20: ac000cc8 748001a1 748001b7 ac000cc8 748001b7 ac000cc8 00000005
82412640
0c40: ac003d20 ac003d54 00000051 ac000c68 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000cc8 to 0xac000d10)
0cc0: ac000db8 00000005 ac000d78 748001b7 748001cd
ac000d78
0ce0: 748001cd ac000d78 00000005 82412640 ac003d20 ac003d54 00000051
ac000d18
0d00: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000d78 to 0xac000dc0)
0d60: ac000e68
00000005
0d80: ac000e28 748001cd 748001e3 ac000e28 748001e3 ac000e28 00000005
82412640
0da0: ac003d20 ac003d54 00000051 ac000dc8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000e28 to 0xac000e70)
0e20: ac000f18 00000005 ac000ed8 748001e3 748001f9
ac000ed8
0e40: 748001f9 ac000ed8 00000005 82412640 ac003d20 ac003d54 00000051
ac000e78
0e60: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000ed8 to 0xac000f20)
0ec0: ac000fc8
00000005
0ee0: ac000f88 748001f9 7480020f ac000f88 7480020f ac000f88 00000005
82412640
0f00: ac003d20 ac003d54 00000051 ac000f28 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac000f88 to 0xac000fd0)
0f80: ac001078 00000005 ac001038 7480020f 74800225
ac001038
0fa0: 74800225 ac001038 00000005 82412640 ac003d20 ac003d54 00000051
ac000fd8
0fc0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001038 to 0xac001080)
1020: ac001128
00000005
1040: ac0010e8 74800225 7480023b ac0010e8 7480023b ac0010e8 00000005
82412640
1060: ac003d20 ac003d54 00000051 ac001088 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0010e8 to 0xac001130)
10e0: ac0011d8 00000005 ac001198 7480023b 74800251
ac001198
1100: 74800251 ac001198 00000005 82412640 ac003d20 ac003d54 00000051
ac001138
1120: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001198 to 0xac0011e0)
1180: ac001288
00000005
11a0: ac001248 74800251 74800267 ac001248 74800267 ac001248 00000005
82412640
11c0: ac003d20 ac003d54 00000051 ac0011e8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001248 to 0xac001290)
1240: ac001338 00000005 ac0012f8 74800267 7480027d
ac0012f8
1260: 7480027d ac0012f8 00000005 82412640 ac003d20 ac003d54 00000051
ac001298
1280: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0012f8 to 0xac001340)
12e0: ac0013e8
00000005
1300: ac0013a8 7480027d 74800293 ac0013a8 74800293 ac0013a8 00000005
82412640
1320: ac003d20 ac003d54 00000051 ac001348 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0013a8 to 0xac0013f0)
13a0: ac001498 00000005 ac001458 74800293 748002a9
ac001458
13c0: 748002a9 ac001458 00000005 82412640 ac003d20 ac003d54 00000051
ac0013f8
13e0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001458 to 0xac0014a0)
1440: ac001548
00000005
1460: ac001508 748002a9 748002bf ac001508 748002bf ac001508 00000005
82412640
1480: ac003d20 ac003d54 00000051 ac0014a8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001508 to 0xac001550)
1500: ac0015f8 00000005 ac0015b8 748002bf 748002d5
ac0015b8
1520: 748002d5 ac0015b8 00000005 82412640 ac003d20 ac003d54 00000051
ac001558
1540: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0015b8 to 0xac001600)
15a0: ac0016a8
00000005
15c0: ac001668 748002d5 748002eb ac001668 748002eb ac001668 00000005
82412640
15e0: ac003d20 ac003d54 00000051 ac001608 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001668 to 0xac0016b0)
1660: ac001758 00000005 ac001718 748002eb 74800301
ac001718
1680: 74800301 ac001718 00000005 82412640 ac003d20 ac003d54 00000051
ac0016b8
16a0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001718 to 0xac001760)
1700: ac001808
00000005
1720: ac0017c8 74800301 74800317 ac0017c8 74800317 ac0017c8 00000005
82412640
1740: ac003d20 ac003d54 00000051 ac001768 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0017c8 to 0xac001810)
17c0: ac0018b8 00000005 ac001878 74800317 7480032d
ac001878
17e0: 7480032d ac001878 00000005 82412640 ac003d20 ac003d54 00000051
ac001818
1800: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001878 to 0xac0018c0)
1860: ac001968
00000005
1880: ac001928 7480032d 74800343 ac001928 74800343 ac001928 00000005
82412640
18a0: ac003d20 ac003d54 00000051 ac0018c8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001928 to 0xac001970)
1920: ac001a18 00000005 ac0019d8 74800343 74800359
ac0019d8
1940: 74800359 ac0019d8 00000005 82412640 ac003d20 ac003d54 00000051
ac001978
1960: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0019d8 to 0xac001a20)
19c0: ac001ac8
00000005
19e0: ac001a88 74800359 7480036f ac001a88 7480036f ac001a88 00000005
82412640
1a00: ac003d20 ac003d54 00000051 ac001a28 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001a88 to 0xac001ad0)
1a80: ac001b78 00000005 ac001b38 7480036f 74800385
ac001b38
1aa0: 74800385 ac001b38 00000005 82412640 ac003d20 ac003d54 00000051
ac001ad8
1ac0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001b38 to 0xac001b80)
1b20: ac001c28
00000005
1b40: ac001be8 74800385 7480039b ac001be8 7480039b ac001be8 00000005
82412640
1b60: ac003d20 ac003d54 00000051 ac001b88 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001be8 to 0xac001c30)
1be0: ac001cd8 00000005 ac001c98 7480039b 748003b1
ac001c98
1c00: 748003b1 ac001c98 00000005 82412640 ac003d20 ac003d54 00000051
ac001c38
1c20: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001c98 to 0xac001ce0)
1c80: ac001d88
00000005
1ca0: ac001d48 748003b1 748003c7 ac001d48 748003c7 ac001d48 00000005
82412640
1cc0: ac003d20 ac003d54 00000051 ac001ce8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001d48 to 0xac001d90)
1d40: ac001e38 00000005 ac001df8 748003c7 748003dd
ac001df8
1d60: 748003dd ac001df8 00000005 82412640 ac003d20 ac003d54 00000051
ac001d98
1d80: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001df8 to 0xac001e40)
1de0: ac001ee8
00000005
1e00: ac001ea8 748003dd 748003f3 ac001ea8 748003f3 ac001ea8 00000005
82412640
1e20: ac003d20 ac003d54 00000051 ac001e48 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001ea8 to 0xac001ef0)
1ea0: ac001f98 00000005 ac001f58 748003f3 74800409
ac001f58
1ec0: 74800409 ac001f58 00000005 82412640 ac003d20 ac003d54 00000051
ac001ef8
1ee0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac001f58 to 0xac001fa0)
1f40: ac002048
00000005
1f60: ac002008 74800409 7480041f ac002008 7480041f ac002008 00000005
82412640
1f80: ac003d20 ac003d54 00000051 ac001fa8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002008 to 0xac002050)
2000: ac0020f8 00000005 ac0020b8 7480041f 74800435
ac0020b8
2020: 74800435 ac0020b8 00000005 82412640 ac003d20 ac003d54 00000051
ac002058
2040: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0020b8 to 0xac002100)
20a0: ac0021a8
00000005
20c0: ac002168 74800435 7480044b ac002168 7480044b ac002168 00000005
82412640
20e0: ac003d20 ac003d54 00000051 ac002108 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002168 to 0xac0021b0)
2160: ac002258 00000005 ac002218 7480044b 74800461
ac002218
2180: 74800461 ac002218 00000005 82412640 ac003d20 ac003d54 00000051
ac0021b8
21a0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002218 to 0xac002260)
2200: ac002308
00000005
2220: ac0022c8 74800461 74800477 ac0022c8 74800477 ac0022c8 00000005
82412640
2240: ac003d20 ac003d54 00000051 ac002268 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0022c8 to 0xac002310)
22c0: ac0023b8 00000005 ac002378 74800477 7480048d
ac002378
22e0: 7480048d ac002378 00000005 82412640 ac003d20 ac003d54 00000051
ac002318
2300: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002378 to 0xac0023c0)
2360: ac002468
00000005
2380: ac002428 7480048d 748004a3 ac002428 748004a3 ac002428 00000005
82412640
23a0: ac003d20 ac003d54 00000051 ac0023c8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002428 to 0xac002470)
2420: ac002518 00000005 ac0024d8 748004a3 748004b9
ac0024d8
2440: 748004b9 ac0024d8 00000005 82412640 ac003d20 ac003d54 00000051
ac002478
2460: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0024d8 to 0xac002520)
24c0: ac0025c8
00000005
24e0: ac002588 748004b9 748004cf ac002588 748004cf ac002588 00000005
82412640
2500: ac003d20 ac003d54 00000051 ac002528 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002588 to 0xac0025d0)
2580: ac002678 00000005 ac002638 748004cf 748004e5
ac002638
25a0: 748004e5 ac002638 00000005 82412640 ac003d20 ac003d54 00000051
ac0025d8
25c0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002638 to 0xac002680)
2620: ac002728
00000005
2640: ac0026e8 748004e5 748004fb ac0026e8 748004fb ac0026e8 00000005
82412640
2660: ac003d20 ac003d54 00000051 ac002688 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0026e8 to 0xac002730)
26e0: ac0027d8 00000005 ac002798 748004fb 74800511
ac002798
2700: 74800511 ac002798 00000005 82412640 ac003d20 ac003d54 00000051
ac002738
2720: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002798 to 0xac0027e0)
2780: ac002888
00000005
27a0: ac002848 74800511 74800527 ac002848 74800527 ac002848 00000005
82412640
27c0: ac003d20 ac003d54 00000051 ac0027e8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002848 to 0xac002890)
2840: ac002938 00000005 ac0028f8 74800527 7480053d
ac0028f8
2860: 7480053d ac0028f8 00000005 82412640 ac003d20 ac003d54 00000051
ac002898
2880: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0028f8 to 0xac002940)
28e0: ac0029e8
00000005
2900: ac0029a8 7480053d 74800553 ac0029a8 74800553 ac0029a8 00000005
82412640
2920: ac003d20 ac003d54 00000051 ac002948 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0029a8 to 0xac0029f0)
29a0: ac002a98 00000005 ac002a58 74800553 74800569
ac002a58
29c0: 74800569 ac002a58 00000005 82412640 ac003d20 ac003d54 00000051
ac0029f8
29e0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002a58 to 0xac002aa0)
2a40: ac002b48
00000005
2a60: ac002b08 74800569 7480057f ac002b08 7480057f ac002b08 00000005
82412640
2a80: ac003d20 ac003d54 00000051 ac002aa8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002b08 to 0xac002b50)
2b00: ac002bf8 00000005 ac002bb8 7480057f 74800595
ac002bb8
2b20: 74800595 ac002bb8 00000005 82412640 ac003d20 ac003d54 00000051
ac002b58
2b40: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002bb8 to 0xac002c00)
2ba0: ac002ca8
00000005
2bc0: ac002c68 74800595 748005ab ac002c68 748005ab ac002c68 00000005
82412640
2be0: ac003d20 ac003d54 00000051 ac002c08 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002c68 to 0xac002cb0)
2c60: ac002d58 00000005 ac002d18 748005ab 748005c1
ac002d18
2c80: 748005c1 ac002d18 00000005 82412640 ac003d20 ac003d54 00000051
ac002cb8
2ca0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002d18 to 0xac002d60)
2d00: ac002e08
00000005
2d20: ac002dc8 748005c1 748005d7 ac002dc8 748005d7 ac002dc8 00000005
82412640
2d40: ac003d20 ac003d54 00000051 ac002d68 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002dc8 to 0xac002e10)
2dc0: ac002eb8 00000005 ac002e78 748005d7 748005ed
ac002e78
2de0: 748005ed ac002e78 00000005 82412640 ac003d20 ac003d54 00000051
ac002e18
2e00: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002e78 to 0xac002ec0)
2e60: ac002f68
00000005
2e80: ac002f28 748005ed 74800603 ac002f28 74800603 ac002f28 00000005
82412640
2ea0: ac003d20 ac003d54 00000051 ac002ec8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002f28 to 0xac002f70)
2f20: ac003018 00000005 ac002fd8 74800603 74800619
ac002fd8
2f40: 74800619 ac002fd8 00000005 82412640 ac003d20 ac003d54 00000051
ac002f78
2f60: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac002fd8 to 0xac003020)
2fc0: ac0030c8
00000005
2fe0: ac003088 74800619 7480062f ac003088 7480062f ac003088 00000005
82412640
3000: ac003d20 ac003d54 00000051 ac003028 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003088 to 0xac0030d0)
3080: ac003178 00000005 ac003138 7480062f 74800645
ac003138
30a0: 74800645 ac003138 00000005 82412640 ac003d20 ac003d54 00000051
ac0030d8
30c0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003138 to 0xac003180)
3120: ac003228
00000005
3140: ac0031e8 74800645 7480065b ac0031e8 7480065b ac0031e8 00000005
82412640
3160: ac003d20 ac003d54 00000051 ac003188 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0031e8 to 0xac003230)
31e0: ac0032d8 00000005 ac003298 7480065b 74800671
ac003298
3200: 74800671 ac003298 00000005 82412640 ac003d20 ac003d54 00000051
ac003238
3220: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003298 to 0xac0032e0)
3280: ac003388
00000005
32a0: ac003348 74800671 74800687 ac003348 74800687 ac003348 00000005
82412640
32c0: ac003d20 ac003d54 00000051 ac0032e8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003348 to 0xac003390)
3340: ac003438 00000005 ac0033f8 74800687 7480069d
ac0033f8
3360: 7480069d ac0033f8 00000005 82412640 ac003d20 ac003d54 00000051
ac003398
3380: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0033f8 to 0xac003440)
33e0: ac0034e8
00000005
3400: ac0034a8 7480069d 748006b3 ac0034a8 748006b3 ac0034a8 00000005
82412640
3420: ac003d20 ac003d54 00000051 ac003448 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0034a8 to 0xac0034f0)
34a0: ac003598 00000005 ac003558 748006b3 748006c9
ac003558
34c0: 748006c9 ac003558 00000005 82412640 ac003d20 ac003d54 00000051
ac0034f8
34e0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003558 to 0xac0035a0)
3540: ac003648
00000005
3560: ac003608 748006c9 748006df ac003608 748006df ac003608 00000005
82412640
3580: ac003d20 ac003d54 00000051 ac0035a8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003608 to 0xac003650)
3600: ac0036f8 00000005 ac0036b8 748006df 748006f5
ac0036b8
3620: 748006f5 ac0036b8 00000005 82412640 ac003d20 ac003d54 00000051
ac003658
3640: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0036b8 to 0xac003700)
36a0: ac0037a8
00000005
36c0: ac003768 748006f5 7480070b ac003768 7480070b ac003768 00000005
82412640
36e0: ac003d20 ac003d54 00000051 ac003708 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003768 to 0xac0037b0)
3760: ac003858 00000005 ac003818 7480070b 74800721
ac003818
3780: 74800721 ac003818 00000005 82412640 ac003d20 ac003d54 00000051
ac0037b8
37a0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003818 to 0xac003860)
3800: ac003908
00000005
3820: ac0038c8 74800721 74800737 ac0038c8 74800737 ac0038c8 00000005
82412640
3840: ac003d20 ac003d54 00000051 ac003868 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac0038c8 to 0xac003910)
38c0: ac0039b8 00000005 ac003978 74800737 7480074d
ac003978
38e0: 7480074d ac003978 00000005 82412640 ac003d20 ac003d54 00000051
ac003918
3900: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003978 to 0xac0039c0)
3960: ac003a68
00000005
3980: ac003a28 7480074d 74800763 ac003a28 74800763 ac003a28 00000005
82412640
39a0: ac003d20 ac003d54 00000051 ac0039c8 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003a28 to 0xac003a70)
3a20: ac003b18 00000005 ac003ad8 74800763 74800779
ac003ad8
3a40: 74800779 ac003ad8 00000005 82412640 ac003d20 ac003d54 00000051
ac003a78
3a60: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003ad8 to 0xac003b20)
3ac0: ac003bc8
00000005
3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
82412640
3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003b88 to 0xac003bd0)
3b80: ac003c78 00000805 ac003c38 7480078f 74800798
ac003c38
3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
ac003bd8
3bc0: 80125c14 80125920 200f0193 ffffffff
__dabt_svc from do_translation_fault+0x30/0x2b0
do_translation_fault from do_DataAbort+0x74/0x1dc
do_DataAbort from __dabt_svc+0x4c/0x80
Exception stack(0xac003c38 to 0xac003c80)
3c20: 9ab7bb00
8652c000
3c40: f1f1f1f1 f3f3f304 833bc800 9ab7bb00 83850900 833bcc88 9ab7bb00
74800798
3c60: ac003d20 ac003d54 00000000 ac003c88 81dceb28 801e7c40 600f0093
ffffffff
__dabt_svc from mm_cid_get+0x5c/0x668
mm_cid_get from __schedule+0xcb4/0x23a4
__schedule from schedule+0x5c/0x258
schedule from do_work_pending+0xbc/0xd04
do_work_pending from slow_work_pending+0xc/0x24
Exception stack(0xac003fb0 to 0xac003ff8)
3fa0: fffffffe 6eda3c80 00000800
000007ff
3fc0: 6eda3b20 6eda3cf8 66daa3fc 0000018d ffffff9c 00000000 00000000
66dc9d80
3fe0: 0000018d 6eda3af8 66dc09cf 66dbff66 600f0030 ffffff9c
Code: e24dd00c e1a031a0 e1a08001 e283345f (e1d320d0)
---[ end trace 0000000000000000 ]---
Crash log with kernel at offending commit (b6506981f880):
8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 00000012
[00000012] *pgd=00000000
Internal error: Oops: 5 [#1] SMP ARM
Modules linked in:
CPU: 1 PID: 1641 Comm: find Tainted: G B 5.16.0-rc1+ #1
Hardware name: ARM-Versatile Express
PC is at __read_once_word_nocheck+0x0/0x8
LR is at unwind_frame+0x3f8/0x874
pc : [<80112b48>] lr : [<80112ffc>] psr: 60000113
sp : 82d67998 ip : 801cff34 fp : 810cc2cc
r10: 00000001 r9 : 00000000 r8 : 82d67aa0
r7 : 00000483 r6 : 00000016 r5 : 82d67a10 r4 : 00000012
r3 : 00000000 r2 : 00000083 r1 : 00000010 r0 : 00000012
Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c5387d Table: 888c806a DAC: 00000051
Register r0 information: non-paged memory
Register r1 information: zero-size pointer
Register r2 information: non-paged memory
Register r3 information: NULL pointer
Register r4 information: non-paged memory
Register r5 information: non-slab/vmalloc memory
Register r6 information: non-paged memory
Register r7 information: non-paged memory
Register r8 information: non-slab/vmalloc memory
Register r9 information: NULL pointer
Register r10 information: non-paged memory
Register r11 information: non-slab/vmalloc memory
Register r12 information: non-slab/vmalloc memory
Process find (pid: 1641, stack limit = 0xf7b89a2f)
Stack: (0x82d67998 to 0x82d68000)
7980: 00004000
00000012
79a0: 82d67adc 82d67ae0 82d67ae8 824fbd40 6f5acf3c ffffc000 82d67ae4
00000000
79c0: 00000001 82d67e74 00000000 80359e68 00002c8b 00002c8b 00000400
000003f7
79e0: 41b58ab3 80fd9554 80112c04 00000000 00000001 00000005 00000000
00000000
7a00: 824fbd40 00000000 824fbd40 00000000 00000003 824fbd40 82d67f40
6f5acfdc
7a20: 824fc3bc 6e86ea3c 0000000b 0000001e 00000051 00000012 8033d154
00000000
7a40: 810c9e28 80100060 00000078 1d5cffa2 00000000 82d67b80 82d67c50
6f5acf54
7a60: 00000008 824fbd40 82d67c50 82d67ce0 00000001 801808cc 00002c8b
ffffc000
7a80: 82d67bac 00000000 008002a9 80112bd0 810cbb48 82d67bfc 810c9e28
80100060
7aa0: 41b58ab3 1d5cffa2 80180650 824fbd40 6f5acf64 824fbd40 00000000
00000000
7ac0: 801c51d8 000000c8 82d67afc 8010dc54 82d67b40 00000000 00000000
0000001e
7ae0: 82d67ec8 8033d154 80c4d2ac 1d5cffa2 0000000b 82d67b80 83158080
801cffb8
7b00: 00000000 00000000 000030ef 000030ef 00000400 0000033b 0000273d
0000000c
7b20: 41b58ab3 80fe3f84 801cff34 00000005 00000000 00000000 82d67c64
00000000
7b40: 00000009 00000040 82d67b98 00000001 6ee00000 82d67fa8 80100060
00000000
7b60: 41b58ab3 80fd9554 80112c04 1d5cffa2 00000000 824fbd40 0000003c
1d5cffa2
7b80: 00000000 83158004 00000800 83158000 81897700 80322d6c 8032466c
80323150
7ba0: 8032165c 801c51d8 8010132c 80134788 80100b10 80c4d2ac 80c4d2ac
00000000
7bc0: 0000b6ee 80194ae0 00003000 00000001 8117bc80 0000000c 00000001
00002954
7be0: 80d2a4e0 00000000 00000000 81245940 0000294d 801bf3c8 6f5acf84
00000001
7c00: 9a9b6388 82d67d80 00000070 0000001d 0000753f 81245180 81245180
00002954
7c20: 81245188 81430b24 9a9bd3c0 00000000 81430b24 801c3ce4 00000cc0
00000012
7c40: 9a9bd430 00000003 801948f4 00000001 00000000 80d2a2e0 00002954
80d2a320
7c60: 81430b24 9a9bd400 8142af20 801c3ee8 00000000 9a9b6388 00000000
00000000
7c80: 00000020 9a9bd3c0 9a9bd3c0 81245180 00000000 81245184 9a9bd3cc
83158000
7ca0: 9ec54b00 8032466c 00000000 80323150 83158000 81897700 801c51d8
9ec54b00
7cc0: 8144eb80 81245180 824fbd44 8032165c 60000113 80c558d4 00000000
81245180
7ce0: 81245940 81245940 824fbd40 00000001 8142af84 82d67d80 81245180
801c51d8
7d00: 00000000 818adf2c 818adf00 80181b10 0000000a 00000000 8142af80
00000000
7d20: 81206ccc 9a9bd400 824fbf54 81430b24 6f5acfac 824fbd48 8142ad80
82d67dc0
7d40: 9a9bd424 8143efe0 812070e4 9a9bd448 9a9bd3c0 00000001 00000000
fffffffb
7d60: 41b58ab3 80fe3708 801c4df4 801da208 824fbd40 00000016 00000000
9a9c0200
7d80: 84847ba0 81a025a0 00000007 00000000 9a9c0200 00091069 00048834
00000000
7da0: 81896980 801cfe54 4f512b00 9a9c0200 00000016 00989680 00000000
00000001
7dc0: 19841000 801ebfd0 00000000 822ce180 00000000 1d5cffa2 81894800
812050a4
7de0: 00000002 00000009 8142a7c0 00000002 824fbd40 81175380 824fbd44
8010132c
7e00: 0000001b 824fbd40 00000100 19841000 81205080 81175358 0000000a
801a454c
7e20: 811752d0 8117b840 8117b840 ffffb03e 81205d00 00404040 824fbf54
80d143c0
7e40: 0000000b 81446da0 19841000 82d67e68 82d67eac 824fc3bc 824fbd40
0000000b
7e60: 0000001e 80134788 80c4d2ac 60000013 ffffffff 80100b10 824fbd40
00000000
7e80: 824fbd40 00000000 00000003 824fbd40 82d67f40 6f5acfdc 824fc3bc
6e86ea3c
7ea0: 0000000b 0000001e 00000051 82d67ec8 8033d154 80c4d2ac 60000013
ffffffff
7ec0: 00000051 8033d144 00000000 6ecff000 00000000 802e8b20 82be3400
00100000
7ee0: 41b58ab3 80ff5ed8 8033d0a0 0006edff 822ce180 00000000 6ee00000
82be3400
7f00: 6e86e4f8 82be3444 00000001 8285f000 8285f008 82be3400 822ce1a8
80197c9c
7f20: 8285f000 822ce180 00000001 8033e988 6edfffff 822ce184 00000cc0
824fbd40
7f40: 824fbf54 1d5cffa2 824fbd40 8285f000 82305500 ffffff9c 6e86e4f8
8033f418
7f60: 00000000 0000000b 0000001e 6e86e4f8 6e86ea3c 6e86e4f8 0000000b
80100284
7f80: 824fbd40 0000000b 0000001e 803409d4 00000000 6e86e4f8 0000000b
66cf7360
7fa0: 00000001 80100060 66cf7360 00000001 6e86e470 6e86e4f8 6e86ea3c
70657267
7fc0: 66cf7360 00000001 6e86e4f8 0000000b 6e86ea3c 00000000 6e86ebd0
0000001e
7fe0: 6e86e47e 6e86e46c 66c68cf5 66c68758 80000030 6e86e470 00000000
00000000
[<80112b48>] (__read_once_word_nocheck) from [<80112ffc>]
(unwind_frame+0x3f8/0x874)
[<80112ffc>] (unwind_frame) from [<8010dc54>] (__save_stack_trace+0x70/0x94)
[<8010dc54>] (__save_stack_trace) from [<801cffb8>]
(stack_trace_save+0x84/0xac)
[<801cffb8>] (stack_trace_save) from [<80322d6c>]
(kasan_set_track+0x2c/0x58)
[<80322d6c>] (kasan_set_track) from [<8032466c>]
(kasan_set_free_info+0x20/0x34)
[<8032466c>] (kasan_set_free_info) from [<80323150>]
(__kasan_slab_free+0xdc/0x108)
[<80323150>] (__kasan_slab_free) from [<8032165c>]
(kmem_cache_free+0x80/0x304)
[<8032165c>] (kmem_cache_free) from [<801c51d8>] (rcu_core+0x3e4/0xc1c)
[<801c51d8>] (rcu_core) from [<8010132c>] (__do_softirq+0x17c/0x4dc)
[<8010132c>] (__do_softirq) from [<80134788>] (irq_exit+0xb4/0xfc)
[<80134788>] (irq_exit) from [<80100b10>] (__irq_svc+0x50/0x68)
Exception stack(0x82d67e78 to 0x82d67ec0)
7e60: 824fbd40
00000000
7e80: 824fbd40 00000000 00000003 824fbd40 82d67f40 6f5acfdc 824fc3bc
6e86ea3c
7ea0: 0000000b 0000001e 00000051 82d67ec8 8033d154 80c4d2ac 60000013
ffffffff
[<80100b10>] (__irq_svc) from [<80c4d2ac>] (__cond_resched+0x0/0x64)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-14 13:19 Crash on armv7-a using KASAN Clement LE GOFFIC
@ 2024-10-15 7:55 ` Linus Walleij
2024-10-15 10:35 ` Clement LE GOFFIC
2024-10-15 10:28 ` Mark Rutland
1 sibling, 1 reply; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 7:55 UTC (permalink / raw)
To: Clement LE GOFFIC
Cc: Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
Hi Clement,
thanks for your report! I looked a bit at it:
On Mon, Oct 14, 2024 at 3:21 PM Clement LE GOFFIC
<clement.legoffic@foss.st.com> wrote:
> I have detected a kernel crash in latest kernel on armv7-a when Kasan is
> enabled.
(...)
> Crash log with recent kernel (v6.12-rc3) :
>
> ~ # Insufficient stack space to handle exception!
The crash looks pretty "expected", as you say you start a lot of
parallel processes
and whoops, you run out of memory on the stack. No software can add more
memory to the machine.
KASAN uses a lot of extra memory for intercepting all memory accesses,
nominally one
extra byte per 8 bytes. This is further restricted by the complex
nature of the virtual
memory space on ARM32.
That said, we increase the size of per-thread storage when using KASAN,
THREAD_SIZE_ORDER is 2 instead of 1. Maybe the interrupt stacks need
to be scaled similarly to manage the increased load?
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-14 13:19 Crash on armv7-a using KASAN Clement LE GOFFIC
2024-10-15 7:55 ` Linus Walleij
@ 2024-10-15 10:28 ` Mark Rutland
2024-10-15 13:51 ` Linus Walleij
1 sibling, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-15 10:28 UTC (permalink / raw)
To: Clement LE GOFFIC, Linus Walleij
Cc: Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> Hello,
Hi,
I think what's happening here is that when switching from prev to next
in the scheduler, we switch to next's mm before we actually switch to
next's register state, and there's a transient window where prev is
executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
into next's mm anywhere, and so inlined KASAN_STACK checks recursively
fault on this until we switch to the overflow stack.
More details on that below.
Linus, are you able to look into this?
> I have detected a kernel crash in latest kernel on armv7-a when Kasan is
> enabled.
> Git bisect points to Ard commit from 2021 :
> b6506981f880 ARM: unwind: support unwinding across multiple stacks
> ardb@kernel.org
>
> This part of code is outside my expertise and I'm unable to propose a fix.
> Below there is a simple way to replicate the crash inside qemu.
> The offending commit is a preliminary patch of a series aimed at managing
> multiple stacks.
> The crash log from the offending commit differs with the crash log from
> latest kernel due to the extra patches in the mentioned series.
>
> Step to reproduce :
> 1. Ensure `qemu-system-arm` and `arm-none-eabi-gcc` are installed on your
> system
>
> 2. Get a filesystem. I found a Debian installer filesystem (publicly
> available):
> $> wget http://ftp.us.debian.org/debian/dists/stable/main/installer-armhf/current/images/cdrom/initrd.gz
>
> 3. Configure and build the kernel with KASAN enabled:
> $> echo 'CONFIG_KASAN=y' > arch/arm/configs/fragment-kasan.config
> $> make ARCH=arm CROSS_COMPILE=arm-none-eabi- vexpress_defconfig
> fragment-kasan.config
> $> make ARCH=arm CROSS_COMPILE=arm-none-eabi- -j16 all
>
> 4. Run the kernel in QEMU:
> $> qemu-system-arm -kernel arch/arm/boot/zImage -machine vexpress-a15 -dtb
> arch/arm/boot/dts/arm/vexpress-v2p-ca15-tc1.dtb -append "console=ttyAMA0"
> -initrd initrd.gz -m 512 -smp 2
>
> 5. When the "Select a language" menu appears, press ESC, in the new menu
> select "Execute a shell", then "Continue"
>
> 6. The crash may appear directly during boot. If not, you can stress the
> system with syscalls by running ten or more executions in parallel of:
> $> find /usr/ -type f -exec grep randomStringToSearch {} \;&
>
> 7. Wait for a kernel crash.
>
> Thank you for looking into this issue. Please let me know if you need any
> further information.
>
> Best regards,
>
> Clément Le Goffic
>
> NB: Two kernel crash dumps are following each other.
>
> Crash log with recent kernel (v6.12-rc3) :
>
> ~ # Insufficient stack space to handle exception!
> Task stack: [0xac000000..0xac004000]
> IRQ stack: [0xa0808000..0xa080c000]
> Overflow stack: [0x82964000..0x82965000]
> Internal error: kernel stack overflow: 0 [#1] SMP ARM
> Modules linked in:
> CPU: 1 UID: 0 PID: 1678 Comm: grep Not tainted 6.12.0-rc3 #2
> Hardware name: ARM-Versatile Express
> PC is at do_translation_fault+0x30/0x2b0
> LR is at do_DataAbort+0x74/0x1dc
> pc : [<80125920>] lr : [<80125c14>] psr: 200f0193
> sp : ac000008 ip : 00000051 fp : ac003d54
> r10: ac003d20 r9 : 82412640 r8 : 00000005
> r7 : ac000068 r6 : 7480002b r5 : ac000068 r4 : 7480002b
> r3 : 74800015 r2 : ac000068 r1 : 00000005 r0 : ac0000a8
> Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
> Control: 10c5387d Table: 88a8406a DAC: 00000051
[...]
> __dabt_svc from do_translation_fault+0x30/0x2b0
> do_translation_fault from do_DataAbort+0x74/0x1dc
> do_DataAbort from __dabt_svc+0x4c/0x80
> Exception stack(0xac003ad8 to 0xac003b20)
> 3ac0: ac003bc8
> 00000005
> 3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
> 82412640
> 3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
> ffffffff
> __dabt_svc from do_translation_fault+0x30/0x2b0
> do_translation_fault from do_DataAbort+0x74/0x1dc
> do_DataAbort from __dabt_svc+0x4c/0x80
> Exception stack(0xac003b88 to 0xac003bd0)
> 3b80: ac003c78 00000805 ac003c38 7480078f 74800798
> ac003c38
> 3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
> ac003bd8
> 3bc0: 80125c14 80125920 200f0193 ffffffff
> __dabt_svc from do_translation_fault+0x30/0x2b0
> do_translation_fault from do_DataAbort+0x74/0x1dc
> do_DataAbort from __dabt_svc+0x4c/0x80
The above frames are the same; whatever the kernel is accessing at
do_translation_fault+0x30 is causing this to go recursive...
I can reproduce this, pretty easily, with a similar enough trace, though
faddr2line isn't happy to give me a line number. The relevant asm is:
| 801272d8 <do_translation_fault>:
| 801272d8: e92d4ff0 push {r4, r5, r6, r7, r8, r9, sl, fp, lr}
| 801272dc: e30f3fff movw r3, #65535 @ 0xffff
| 801272e0: e3463edf movt r3, #28383 @ 0x6edf
| 801272e4: e24dd00c sub sp, sp, #12
| 801272e8: e1500003 cmp r0, r3
| 801272ec: 9a00008b bls 80127520 <do_translation_fault+0x248>
| 801272f0: e1a04000 mov r4, r0
| 801272f4: e2820040 add r0, r2, #64 @ 0x40
| 801272f8: e1a05002 mov r5, r2
| 801272fc: e1a07001 mov r7, r1
| 80127300: e1a031a0 lsr r3, r0, #3
| 80127304: e283345f add r3, r3, #1593835520 @ 0x5f000000
| 80127308: e1d320d0 ldrsb r2, [r3]
... with the LDRSB at 80127308 being the faulting instruction.
It looks like that's an access of the shadow, given 0x5f000000 is a shadow
offset per arch/arm/Kconfig:
| 1052 config KASAN_SHADOW_OFFSET
| 1053 hex
| 1054 depends on KASAN
| 1055 default 0x1f000000 if PAGE_OFFSET=0x40000000
| 1056 default 0x5f000000 if PAGE_OFFSET=0x80000000
| 1057 default 0x9f000000 if PAGE_OFFSET=0xC0000000
| 1058 default 0x8f000000 if PAGE_OFFSET=0xB0000000
| 1059 default 0xffffffff
Using addr2line on that address gives me:
| do_translation_fault
| /home/mark/src/linux/arch/arm/mm/fault.c:477
... with the nearby context being:
| 474 if (addr < TASK_SIZE)
| 475 return do_page_fault(addr, fsr, regs);
| 476
| 477 if (user_mode(regs))
| 478 goto bad_area;
... so it looks like a shadow access for 'regs' is faulting. Looking at the
asm, that appears to be the case; with the redundant asm removed it clearly
calculates a shadow address for the argument in r2 (which is 'regs').
| 801272f4: e2820040 add r0, r2, #64 @ 0x40
| 80127300: e1a031a0 lsr r3, r0, #3
| 80127304: e283345f add r3, r3, #1593835520 @ 0x5f000000
| 80127308: e1d320d0 ldrsb r2, [r3]
... so evidently the KASAN shadow for the stack is not mapped at this point for
some reason, but when we get to the overflow stack all is well.
> Exception stack(0xac003c38 to 0xac003c80)
> 3c20: 9ab7bb00
> 8652c000
> 3c40: f1f1f1f1 f3f3f304 833bc800 9ab7bb00 83850900 833bcc88 9ab7bb00
> 74800798
> 3c60: ac003d20 ac003d54 00000000 ac003c88 81dceb28 801e7c40 600f0093
> ffffffff
> __dabt_svc from mm_cid_get+0x5c/0x668
> mm_cid_get from __schedule+0xcb4/0x23a4
The initial fault was taken from mm_cid_get+0x5c. In my compiled version I see
this from mm_cid_get+0x60/0x61c, where the relevant asm is:
| 801ead54 <mm_cid_get>:
| 801ead54: e92d4ff0 push {r4, r5, r6, r7, r8, r9, sl, fp, lr}
| 801ead58: e3082ab3 movw r2, #35507 @ 0x8ab3
| 801ead5c: e34421b5 movt r2, #16821 @ 0x41b5
| 801ead60: e28db020 add fp, sp, #32
| 801ead64: e24dd0a4 sub sp, sp, #164 @ 0xa4
| 801ead68: e24b3024 sub r3, fp, #36 @ 0x24
| 801ead6c: e3c3a01f bic sl, r3, #31
| 801ead70: e1a08000 mov r8, r0
| 801ead74: e24a3060 sub r3, sl, #96 @ 0x60
| 801ead78: e50b10ac str r1, [fp, #-172] @ 0xffffff54
| 801ead7c: e1a091a3 lsr r9, r3, #3
| 801ead80: e50a2060 str r2, [sl, #-96] @ 0xffffffa0
| 801ead84: e289c45f add ip, r9, #1593835520 @ 0x5f000000
| 801ead88: e30b2f38 movw r2, #48952 @ 0xbf38
| 801ead8c: e3482216 movt r2, #33302 @ 0x8216
| 801ead90: e50bc0b0 str ip, [fp, #-176] @ 0xffffff50
| 801ead94: e30f31f1 movw r3, #61937 @ 0xf1f1
| 801ead98: e34f31f1 movt r3, #61937 @ 0xf1f1
| 801ead9c: e50a205c str r2, [sl, #-92] @ 0xffffffa4
| 801eada0: e30a2d54 movw r2, #44372 @ 0xad54
| 801eada4: e348201e movt r2, #32798 @ 0x801e
| 801eada8: e50a2058 str r2, [sl, #-88] @ 0xffffffa8
| 801eadac: e2812064 add r2, r1, #100 @ 0x64
| 801eadb0: e50b20c4 str r2, [fp, #-196] @ 0xffffff3c
| 801eadb4: e58c3000 str r3, [ip]
... with the STR at 801eadb4 being the faulting instruction, and that also
appears to be a faulting KASAN shadow access given ip has that same 0x5f000000
offset.
Per addr2line, that's:
| mm_cid_get
| /home/mark/src/linux/kernel/sched/sched.h:3691
Where the surrounding context is:
| 3690 static inline int mm_cid_get(struct rq *rq, struct mm_struct *mm)
| 3691 {
| 3692 struct mm_cid __percpu *pcpu_cid = mm->pcpu_cid;
| 3693 struct cpumask *cpumask;
| 3694 int cid;
| 3695
| 3696 lockdep_assert_rq_held(rq);
| 3697 cpumask = mm_cidmask(mm);
| 3698 cid = __this_cpu_read(pcpu_cid->cid);
Where IIUC that caller is switch_mm_cid(), which has:
| 3760 if (next->mm_cid_active)
| 3761 next->last_mm_cid = next->mm_cid = mm_cid_get(rq, next->mm);
Removing the redundant instructions, the important sequence is:
| 801ead54 <mm_cid_get>:
| 801ead54: e92d4ff0 push {r4, r5, r6, r7, r8, r9, sl, fp, lr}
| 801ead60: e28db020 add fp, sp, #32
| 801ead68: e24b3024 sub r3, fp, #36 @ 0x24
| 801ead6c: e3c3a01f bic sl, r3, #31
| 801ead74: e24a3060 sub r3, sl, #96 @ 0x60
| 801ead7c: e1a091a3 lsr r9, r3, #3
| 801ead84: e289c45f add ip, r9, #1593835520 @ 0x5f000000
| 801ead94: e30f31f1 movw r3, #61937 @ 0xf1f1
| 801ead98: e34f31f1 movt r3, #61937 @ 0xf1f1
| 801eadb4: e58c3000 str r3, [ip]
... where IIUC it's initialising the stack shadow for local variables.
So again, it looks like the KASAN shadow for the stack is not mapped...
It looks like this is called under context_switch(), where we do:
/* switch_mm_cid() requires the memory barriers above. */
switch_mm_cid(rq, prev, next);
prepare_lock_switch(rq, next, rf);
/* Here we just switch the register state and the stack. */
switch_to(prev, next, prev);
barrier();
... so we're using the new task's mm, but still executing in the context of the
old task (and using its stack). I suspect the new task's mm doesn't have the
old task's stack shadow mapped in, and AFAICT we don't map that in explicitly
anywhere before we switch to the new mm.
Linus, can you look into that?
[...]
> Crash log with kernel at offending commit (b6506981f880):
>
> 8<--- cut here ---
> Unable to handle kernel NULL pointer dereference at virtual address 00000012
> [00000012] *pgd=00000000
> Internal error: Oops: 5 [#1] SMP ARM
> Modules linked in:
> CPU: 1 PID: 1641 Comm: find Tainted: G B 5.16.0-rc1+ #1
> Hardware name: ARM-Versatile Express
> PC is at __read_once_word_nocheck+0x0/0x8
> LR is at unwind_frame+0x3f8/0x874
This looks to be a distinct issue; it looks like we explicitly access an offset
from NULL, and this looks like it'd be fp->pc.
There have been a number of fixes since commit b6506981f880, and this might be
one of those which has been fixed already.
Mark.
> pc : [<80112b48>] lr : [<80112ffc>] psr: 60000113
> sp : 82d67998 ip : 801cff34 fp : 810cc2cc
> r10: 00000001 r9 : 00000000 r8 : 82d67aa0
> r7 : 00000483 r6 : 00000016 r5 : 82d67a10 r4 : 00000012
> r3 : 00000000 r2 : 00000083 r1 : 00000010 r0 : 00000012
> Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
> Control: 10c5387d Table: 888c806a DAC: 00000051
> Register r0 information: non-paged memory
> Register r1 information: zero-size pointer
> Register r2 information: non-paged memory
> Register r3 information: NULL pointer
> Register r4 information: non-paged memory
> Register r5 information: non-slab/vmalloc memory
> Register r6 information: non-paged memory
> Register r7 information: non-paged memory
> Register r8 information: non-slab/vmalloc memory
> Register r9 information: NULL pointer
> Register r10 information: non-paged memory
> Register r11 information: non-slab/vmalloc memory
> Register r12 information: non-slab/vmalloc memory
> Process find (pid: 1641, stack limit = 0xf7b89a2f)
> Stack: (0x82d67998 to 0x82d68000)
> 7980: 00004000
> 00000012
> 79a0: 82d67adc 82d67ae0 82d67ae8 824fbd40 6f5acf3c ffffc000 82d67ae4
> 00000000
> 79c0: 00000001 82d67e74 00000000 80359e68 00002c8b 00002c8b 00000400
> 000003f7
> 79e0: 41b58ab3 80fd9554 80112c04 00000000 00000001 00000005 00000000
> 00000000
> 7a00: 824fbd40 00000000 824fbd40 00000000 00000003 824fbd40 82d67f40
> 6f5acfdc
> 7a20: 824fc3bc 6e86ea3c 0000000b 0000001e 00000051 00000012 8033d154
> 00000000
> 7a40: 810c9e28 80100060 00000078 1d5cffa2 00000000 82d67b80 82d67c50
> 6f5acf54
> 7a60: 00000008 824fbd40 82d67c50 82d67ce0 00000001 801808cc 00002c8b
> ffffc000
> 7a80: 82d67bac 00000000 008002a9 80112bd0 810cbb48 82d67bfc 810c9e28
> 80100060
> 7aa0: 41b58ab3 1d5cffa2 80180650 824fbd40 6f5acf64 824fbd40 00000000
> 00000000
> 7ac0: 801c51d8 000000c8 82d67afc 8010dc54 82d67b40 00000000 00000000
> 0000001e
> 7ae0: 82d67ec8 8033d154 80c4d2ac 1d5cffa2 0000000b 82d67b80 83158080
> 801cffb8
> 7b00: 00000000 00000000 000030ef 000030ef 00000400 0000033b 0000273d
> 0000000c
> 7b20: 41b58ab3 80fe3f84 801cff34 00000005 00000000 00000000 82d67c64
> 00000000
> 7b40: 00000009 00000040 82d67b98 00000001 6ee00000 82d67fa8 80100060
> 00000000
> 7b60: 41b58ab3 80fd9554 80112c04 1d5cffa2 00000000 824fbd40 0000003c
> 1d5cffa2
> 7b80: 00000000 83158004 00000800 83158000 81897700 80322d6c 8032466c
> 80323150
> 7ba0: 8032165c 801c51d8 8010132c 80134788 80100b10 80c4d2ac 80c4d2ac
> 00000000
> 7bc0: 0000b6ee 80194ae0 00003000 00000001 8117bc80 0000000c 00000001
> 00002954
> 7be0: 80d2a4e0 00000000 00000000 81245940 0000294d 801bf3c8 6f5acf84
> 00000001
> 7c00: 9a9b6388 82d67d80 00000070 0000001d 0000753f 81245180 81245180
> 00002954
> 7c20: 81245188 81430b24 9a9bd3c0 00000000 81430b24 801c3ce4 00000cc0
> 00000012
> 7c40: 9a9bd430 00000003 801948f4 00000001 00000000 80d2a2e0 00002954
> 80d2a320
> 7c60: 81430b24 9a9bd400 8142af20 801c3ee8 00000000 9a9b6388 00000000
> 00000000
> 7c80: 00000020 9a9bd3c0 9a9bd3c0 81245180 00000000 81245184 9a9bd3cc
> 83158000
> 7ca0: 9ec54b00 8032466c 00000000 80323150 83158000 81897700 801c51d8
> 9ec54b00
> 7cc0: 8144eb80 81245180 824fbd44 8032165c 60000113 80c558d4 00000000
> 81245180
> 7ce0: 81245940 81245940 824fbd40 00000001 8142af84 82d67d80 81245180
> 801c51d8
> 7d00: 00000000 818adf2c 818adf00 80181b10 0000000a 00000000 8142af80
> 00000000
> 7d20: 81206ccc 9a9bd400 824fbf54 81430b24 6f5acfac 824fbd48 8142ad80
> 82d67dc0
> 7d40: 9a9bd424 8143efe0 812070e4 9a9bd448 9a9bd3c0 00000001 00000000
> fffffffb
> 7d60: 41b58ab3 80fe3708 801c4df4 801da208 824fbd40 00000016 00000000
> 9a9c0200
> 7d80: 84847ba0 81a025a0 00000007 00000000 9a9c0200 00091069 00048834
> 00000000
> 7da0: 81896980 801cfe54 4f512b00 9a9c0200 00000016 00989680 00000000
> 00000001
> 7dc0: 19841000 801ebfd0 00000000 822ce180 00000000 1d5cffa2 81894800
> 812050a4
> 7de0: 00000002 00000009 8142a7c0 00000002 824fbd40 81175380 824fbd44
> 8010132c
> 7e00: 0000001b 824fbd40 00000100 19841000 81205080 81175358 0000000a
> 801a454c
> 7e20: 811752d0 8117b840 8117b840 ffffb03e 81205d00 00404040 824fbf54
> 80d143c0
> 7e40: 0000000b 81446da0 19841000 82d67e68 82d67eac 824fc3bc 824fbd40
> 0000000b
> 7e60: 0000001e 80134788 80c4d2ac 60000013 ffffffff 80100b10 824fbd40
> 00000000
> 7e80: 824fbd40 00000000 00000003 824fbd40 82d67f40 6f5acfdc 824fc3bc
> 6e86ea3c
> 7ea0: 0000000b 0000001e 00000051 82d67ec8 8033d154 80c4d2ac 60000013
> ffffffff
> 7ec0: 00000051 8033d144 00000000 6ecff000 00000000 802e8b20 82be3400
> 00100000
> 7ee0: 41b58ab3 80ff5ed8 8033d0a0 0006edff 822ce180 00000000 6ee00000
> 82be3400
> 7f00: 6e86e4f8 82be3444 00000001 8285f000 8285f008 82be3400 822ce1a8
> 80197c9c
> 7f20: 8285f000 822ce180 00000001 8033e988 6edfffff 822ce184 00000cc0
> 824fbd40
> 7f40: 824fbf54 1d5cffa2 824fbd40 8285f000 82305500 ffffff9c 6e86e4f8
> 8033f418
> 7f60: 00000000 0000000b 0000001e 6e86e4f8 6e86ea3c 6e86e4f8 0000000b
> 80100284
> 7f80: 824fbd40 0000000b 0000001e 803409d4 00000000 6e86e4f8 0000000b
> 66cf7360
> 7fa0: 00000001 80100060 66cf7360 00000001 6e86e470 6e86e4f8 6e86ea3c
> 70657267
> 7fc0: 66cf7360 00000001 6e86e4f8 0000000b 6e86ea3c 00000000 6e86ebd0
> 0000001e
> 7fe0: 6e86e47e 6e86e46c 66c68cf5 66c68758 80000030 6e86e470 00000000
> 00000000
> [<80112b48>] (__read_once_word_nocheck) from [<80112ffc>]
> (unwind_frame+0x3f8/0x874)
> [<80112ffc>] (unwind_frame) from [<8010dc54>] (__save_stack_trace+0x70/0x94)
> [<8010dc54>] (__save_stack_trace) from [<801cffb8>]
> (stack_trace_save+0x84/0xac)
> [<801cffb8>] (stack_trace_save) from [<80322d6c>]
> (kasan_set_track+0x2c/0x58)
> [<80322d6c>] (kasan_set_track) from [<8032466c>]
> (kasan_set_free_info+0x20/0x34)
> [<8032466c>] (kasan_set_free_info) from [<80323150>]
> (__kasan_slab_free+0xdc/0x108)
> [<80323150>] (__kasan_slab_free) from [<8032165c>]
> (kmem_cache_free+0x80/0x304)
> [<8032165c>] (kmem_cache_free) from [<801c51d8>] (rcu_core+0x3e4/0xc1c)
> [<801c51d8>] (rcu_core) from [<8010132c>] (__do_softirq+0x17c/0x4dc)
> [<8010132c>] (__do_softirq) from [<80134788>] (irq_exit+0xb4/0xfc)
> [<80134788>] (irq_exit) from [<80100b10>] (__irq_svc+0x50/0x68)
> Exception stack(0x82d67e78 to 0x82d67ec0)
> 7e60: 824fbd40
> 00000000
> 7e80: 824fbd40 00000000 00000003 824fbd40 82d67f40 6f5acfdc 824fc3bc
> 6e86ea3c
> 7ea0: 0000000b 0000001e 00000051 82d67ec8 8033d154 80c4d2ac 60000013
> ffffffff
> [<80100b10>] (__irq_svc) from [<80c4d2ac>] (__cond_resched+0x0/0x64)
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 7:55 ` Linus Walleij
@ 2024-10-15 10:35 ` Clement LE GOFFIC
0 siblings, 0 replies; 21+ messages in thread
From: Clement LE GOFFIC @ 2024-10-15 10:35 UTC (permalink / raw)
To: Linus Walleij
Cc: Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
On 10/15/24 09:55, Linus Walleij wrote:> On Mon, Oct 14, 2024 at 3:21 PM
Clement LE GOFFIC
> <clement.legoffic@foss.st.com> wrote:
>
>> I have detected a kernel crash in latest kernel on armv7-a when Kasan is
>> enabled.
> (...)
>> Crash log with recent kernel (v6.12-rc3) :
>>
>> ~ # Insufficient stack space to handle exception!
>
> The crash looks pretty "expected", as you say you start a lot of
> parallel processes
> and whoops, you run out of memory on the stack. No software can add more
> memory to the machine.
>
> KASAN uses a lot of extra memory for intercepting all memory accesses,
> nominally one
> extra byte per 8 bytes. This is further restricted by the complex
> nature of the virtual
> memory space on ARM32.
>
> That said, we increase the size of per-thread storage when using KASAN,
> THREAD_SIZE_ORDER is 2 instead of 1. Maybe the interrupt stacks need
> to be scaled similarly to manage the increased load?
Hi Linus,
Thanks for your reply.
Once I found the issue in latest kernel, with Antonio we firstly tried
to increase the stack size, but it kept crashing.
Then we identified the offending commit and we noticed that it does not
change at all the stack layout.
In fact Ard patch only modifies the way to unwind frames for backtrace.
As far as we understand, KASAN enabled causes the generation of several
backtraces (hashed) on alloc and free to track all the allocation.
The offending commit seams responsible of the crash due to incorrect unwind.
Best regards,
Clément
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 10:28 ` Mark Rutland
@ 2024-10-15 13:51 ` Linus Walleij
2024-10-15 14:00 ` Mark Rutland
0 siblings, 1 reply; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 13:51 UTC (permalink / raw)
To: Mark Rutland
Cc: Clement LE GOFFIC, Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> I think what's happening here is that when switching from prev to next
> in the scheduler, we switch to next's mm before we actually switch to
> next's register state, and there's a transient window where prev is
> executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> fault on this until we switch to the overflow stack.
Oh my, that's pretty advanced. Well spotted!
So it has nothing to do with Ards commit, correlation does not
imply causation.
> More details on that below.
>
> Linus, are you able to look into this?
Of course, I'm trying to reproduce the bug.
> > __dabt_svc from do_translation_fault+0x30/0x2b0
> > do_translation_fault from do_DataAbort+0x74/0x1dc
> > do_DataAbort from __dabt_svc+0x4c/0x80
> > Exception stack(0xac003ad8 to 0xac003b20)
> > 3ac0: ac003bc8
> > 00000005
> > 3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
> > 82412640
> > 3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
> > ffffffff
> > __dabt_svc from do_translation_fault+0x30/0x2b0
> > do_translation_fault from do_DataAbort+0x74/0x1dc
> > do_DataAbort from __dabt_svc+0x4c/0x80
> > Exception stack(0xac003b88 to 0xac003bd0)
> > 3b80: ac003c78 00000805 ac003c38 7480078f 74800798
> > ac003c38
> > 3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
> > ac003bd8
> > 3bc0: 80125c14 80125920 200f0193 ffffffff
> > __dabt_svc from do_translation_fault+0x30/0x2b0
> > do_translation_fault from do_DataAbort+0x74/0x1dc
> > do_DataAbort from __dabt_svc+0x4c/0x80
>
> The above frames are the same; whatever the kernel is accessing at
> do_translation_fault+0x30 is causing this to go recursive...
>
> I can reproduce this, pretty easily, with a similar enough trace, though
> faddr2line isn't happy to give me a line number.
Did you reproduce it the same way with a few find /?
I am trying to reproduce it and failing :/
(Using Torvald's HEAD)
This is my config:
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
CONFIG_KASAN=y
CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
CONFIG_KASAN_GENERIC=y
CONFIG_KASAN_OUTLINE=y
# CONFIG_KASAN_INLINE is not set
# CONFIG_KASAN_STACK is not set
# CONFIG_KASAN_VMALLOC is not set
# CONFIG_KASAN_EXTRA_INFO is not set
Do you use more KASAN?
Then I run:
${QEMU} -M vexpress-a15 -m 512M -no-reboot -smp cpus=2 -kernel
${ZIMAGE} -dtb ${DTB} -append "root=/dev/mmcblk0 rw roottype=ext4
console=ttyAMA0" -serial stdio -drive
if=sd,driver=raw,cache=writeback,file=./arch_rootfs.ext4
This is a rootfs with Debian.
Then I fork a few find /|grep fnord > /dev/null &
root@vexpress:~# find / |grep fnord > /dev/null &
[1] 554
root@vexpress:~# find / |grep fnord > /dev/null &
[2] 556
root@vexpress:~# find / |grep fnord > /dev/null &
[3] 558
root@vexpress:~# find / |grep fnord > /dev/null &
[4] 560
root@vexpress:~# find / |grep fnord > /dev/null &
[5] 562
root@vexpress:~# find / |grep fnord > /dev/null &
[6] 564
root@vexpress:~# find / |grep fnord > /dev/null &
[7] 566
root@vexpress:~# find / |grep fnord > /dev/null &
[8] 568
root@vexpress:~# find / |grep fnord > /dev/null &
[9] 570
root@vexpress:~# find / |grep fnord > /dev/null &
[10] 572
root@vexpress:~# find / |grep fnord > /dev/null &
[11] 574
root@vexpress:~# find / |grep fnord > /dev/null &
[12] 576
root@vexpress:~# find / |grep fnord > /dev/null &
[13] 578
root@vexpress:~# find / |grep fnord > /dev/null &
[14] 580
root@vexpress:~# find / |grep fnord > /dev/null &
[15] 582
root@vexpress:~# find / |grep fnord > /dev/null &
[16] 584
root@vexpress:~# find / |grep fnord > /dev/null &
[17] 586
root@vexpress:~# find / |grep fnord > /dev/null &
^[[A[18] 588
root@vexpress:~# find / |grep fnord > /dev/null &
^[[A[19] 590
root@vexpress:~# find / |grep fnord > /dev/null &
[20] 592
root@vexpress:~#
root@vexpress:~# ps
PID TTY TIME CMD
291 ttyAMA0 00:00:02 login
550 ttyAMA0 00:00:01 bash
553 ttyAMA0 00:00:06 find
554 ttyAMA0 00:00:00 grep
555 ttyAMA0 00:00:04 find
556 ttyAMA0 00:00:00 grep
557 ttyAMA0 00:00:03 find
558 ttyAMA0 00:00:00 grep
559 ttyAMA0 00:00:03 find
560 ttyAMA0 00:00:00 grep
561 ttyAMA0 00:00:03 find
562 ttyAMA0 00:00:00 grep
563 ttyAMA0 00:00:02 find
564 ttyAMA0 00:00:00 grep
565 ttyAMA0 00:00:02 find
566 ttyAMA0 00:00:00 grep
567 ttyAMA0 00:00:02 find
568 ttyAMA0 00:00:00 grep
569 ttyAMA0 00:00:02 find
570 ttyAMA0 00:00:00 grep
571 ttyAMA0 00:00:02 find
572 ttyAMA0 00:00:00 grep
573 ttyAMA0 00:00:01 find
574 ttyAMA0 00:00:00 grep
575 ttyAMA0 00:00:01 find
576 ttyAMA0 00:00:00 grep
577 ttyAMA0 00:00:01 find
578 ttyAMA0 00:00:00 grep
579 ttyAMA0 00:00:01 find
580 ttyAMA0 00:00:00 grep
581 ttyAMA0 00:00:01 find
582 ttyAMA0 00:00:00 grep
583 ttyAMA0 00:00:01 find
584 ttyAMA0 00:00:00 grep
585 ttyAMA0 00:00:01 find
586 ttyAMA0 00:00:00 grep
587 ttyAMA0 00:00:01 find
588 ttyAMA0 00:00:00 grep
589 ttyAMA0 00:00:01 find
590 ttyAMA0 00:00:00 grep
591 ttyAMA0 00:00:01 find
592 ttyAMA0 00:00:00 grep
593 ttyAMA0 00:00:01 ps
root@vexpress:~#
This refused to crash.
Then I recompiled with GCC as I was using LLVM CLANG. But
same non-problem: no crash.
> The relevant asm is:
(...)
> ... so we're using the new task's mm, but still executing in the context of the
> old task (and using its stack). I suspect the new task's mm doesn't have the
> old task's stack shadow mapped in, and AFAICT we don't map that in explicitly
> anywhere before we switch to the new mm.
>
> Linus, can you look into that?
Yeah it looks like a spot-on identification of the problem, I can try to
think about how we could fix this if I can reproduce it, I keep trying
to provoke the crash :/
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 13:51 ` Linus Walleij
@ 2024-10-15 14:00 ` Mark Rutland
2024-10-15 14:22 ` Ard Biesheuvel
2024-10-15 14:40 ` Linus Walleij
0 siblings, 2 replies; 21+ messages in thread
From: Mark Rutland @ 2024-10-15 14:00 UTC (permalink / raw)
To: Linus Walleij
Cc: Clement LE GOFFIC, Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
>
> > I think what's happening here is that when switching from prev to next
> > in the scheduler, we switch to next's mm before we actually switch to
> > next's register state, and there's a transient window where prev is
> > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > fault on this until we switch to the overflow stack.
>
> Oh my, that's pretty advanced. Well spotted!
> So it has nothing to do with Ards commit, correlation does not
> imply causation.
>
> > More details on that below.
> >
> > Linus, are you able to look into this?
>
> Of course, I'm trying to reproduce the bug.
>
> > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > do_DataAbort from __dabt_svc+0x4c/0x80
> > > Exception stack(0xac003ad8 to 0xac003b20)
> > > 3ac0: ac003bc8
> > > 00000005
> > > 3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
> > > 82412640
> > > 3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
> > > ffffffff
> > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > do_DataAbort from __dabt_svc+0x4c/0x80
> > > Exception stack(0xac003b88 to 0xac003bd0)
> > > 3b80: ac003c78 00000805 ac003c38 7480078f 74800798
> > > ac003c38
> > > 3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
> > > ac003bd8
> > > 3bc0: 80125c14 80125920 200f0193 ffffffff
> > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > do_DataAbort from __dabt_svc+0x4c/0x80
> >
> > The above frames are the same; whatever the kernel is accessing at
> > do_translation_fault+0x30 is causing this to go recursive...
> >
> > I can reproduce this, pretty easily, with a similar enough trace, though
> > faddr2line isn't happy to give me a line number.
>
> Did you reproduce it the same way with a few find /?
>
> I am trying to reproduce it and failing :/
> (Using Torvald's HEAD)
>
> This is my config:
>
> CONFIG_HAVE_ARCH_KASAN=y
> CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
> CONFIG_CC_HAS_KASAN_GENERIC=y
> CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
> CONFIG_KASAN=y
> CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
> CONFIG_KASAN_GENERIC=y
> CONFIG_KASAN_OUTLINE=y
> # CONFIG_KASAN_INLINE is not set
> # CONFIG_KASAN_STACK is not set
> # CONFIG_KASAN_VMALLOC is not set
> # CONFIG_KASAN_EXTRA_INFO is not set
>
> Do you use more KASAN?
I used a config per Clement's instructions, i.e.
| [mark@lakrids:~/src/linux]% git checkout v6.12-rc3
| HEAD is now at 8e929cb546ee4 Linux 6.12-rc3
| [mark@lakrids:~/src/linux]% git clean -qfdx
| [mark@lakrids:~/src/linux]% echo 'CONFIG_KASAN=y' > arch/arm/configs/fragment-kasan.config
| [mark@lakrids:~/src/linux]% usekorg 14.2.0 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig fragment-kasan.config
| HOSTCC scripts/basic/fixdep
| HOSTCC scripts/kconfig/conf.o
| HOSTCC scripts/kconfig/confdata.o
| HOSTCC scripts/kconfig/expr.o
| LEX scripts/kconfig/lexer.lex.c
| YACC scripts/kconfig/parser.tab.[ch]
| HOSTCC scripts/kconfig/lexer.lex.o
| HOSTCC scripts/kconfig/menu.o
| HOSTCC scripts/kconfig/parser.tab.o
| HOSTCC scripts/kconfig/preprocess.o
| HOSTCC scripts/kconfig/symbol.o
| HOSTCC scripts/kconfig/util.o
| HOSTLD scripts/kconfig/conf
| #
| # configuration written to .config
| #
| Using .config as base
| Merging ./arch/arm/configs/fragment-kasan.config
| Value of CONFIG_KASAN is redefined by fragment ./arch/arm/configs/fragment-kasan.config:
| Previous value: # CONFIG_KASAN is not set
| New value: CONFIG_KASAN=y
|
| #
| # merged configuration written to .config (needs make)
| #
| #
| # configuration written to .config
| #
| [mark@lakrids:~/src/linux]% grep KASAN .config
| CONFIG_KASAN_SHADOW_OFFSET=0x5f000000
| CONFIG_HAVE_ARCH_KASAN=y
| CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
| CONFIG_CC_HAS_KASAN_GENERIC=y
| CONFIG_KASAN=y
| CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
| CONFIG_KASAN_GENERIC=y
| # CONFIG_KASAN_OUTLINE is not set
| CONFIG_KASAN_INLINE=y
| CONFIG_KASAN_STACK=y
| CONFIG_KASAN_VMALLOC=y
| # CONFIG_KASAN_MODULE_TEST is not set
| # CONFIG_KASAN_EXTRA_INFO is not set
IIUC you'll need CONFIG_KASAN_INLINE=y, CONFIG_KASAN_STACK=y, and
CONFIG_KASAN_VMALLOC=y.
With that, just booting and tapping a few keys was sufficient to trigger
a crash using Debian the initrd Clement linked.
[...]
> > The relevant asm is:
> (...)
> > ... so we're using the new task's mm, but still executing in the context of the
> > old task (and using its stack). I suspect the new task's mm doesn't have the
> > old task's stack shadow mapped in, and AFAICT we don't map that in explicitly
> > anywhere before we switch to the new mm.
> >
> > Linus, can you look into that?
>
> Yeah it looks like a spot-on identification of the problem, I can try to
> think about how we could fix this if I can reproduce it, I keep trying
> to provoke the crash :/
It's a bit grotty -- AFAICT you'd either need to prefault in the
specific part of the vmalloc space when switching tasks, or we'd need to
preallocate all the shared vmalloc tables at the start of time so that
they're always up-to-date.
While we could disable KASAN_STACK, that's only going to mask the
problem until this happens for any other vmalloc shadow...
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:00 ` Mark Rutland
@ 2024-10-15 14:22 ` Ard Biesheuvel
2024-10-15 14:35 ` Mark Rutland
2024-10-15 14:40 ` Linus Walleij
1 sibling, 1 reply; 21+ messages in thread
From: Ard Biesheuvel @ 2024-10-15 14:22 UTC (permalink / raw)
To: Mark Rutland
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> >
> > > I think what's happening here is that when switching from prev to next
> > > in the scheduler, we switch to next's mm before we actually switch to
> > > next's register state, and there's a transient window where prev is
> > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > fault on this until we switch to the overflow stack.
> >
> > Oh my, that's pretty advanced. Well spotted!
> > So it has nothing to do with Ards commit, correlation does not
> > imply causation.
> >
> > > More details on that below.
> > >
> > > Linus, are you able to look into this?
> >
> > Of course, I'm trying to reproduce the bug.
> >
> > > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > > do_DataAbort from __dabt_svc+0x4c/0x80
> > > > Exception stack(0xac003ad8 to 0xac003b20)
> > > > 3ac0: ac003bc8
> > > > 00000005
> > > > 3ae0: ac003b88 74800779 7480078f ac003b88 7480078f ac003b88 00000005
> > > > 82412640
> > > > 3b00: ac003d20 ac003d54 00000051 ac003b28 80125c14 80125920 200f0193
> > > > ffffffff
> > > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > > do_DataAbort from __dabt_svc+0x4c/0x80
> > > > Exception stack(0xac003b88 to 0xac003bd0)
> > > > 3b80: ac003c78 00000805 ac003c38 7480078f 74800798
> > > > ac003c38
> > > > 3ba0: 74800798 ac003c38 00000805 82412640 ac003d20 ac003d54 00000051
> > > > ac003bd8
> > > > 3bc0: 80125c14 80125920 200f0193 ffffffff
> > > > __dabt_svc from do_translation_fault+0x30/0x2b0
> > > > do_translation_fault from do_DataAbort+0x74/0x1dc
> > > > do_DataAbort from __dabt_svc+0x4c/0x80
> > >
> > > The above frames are the same; whatever the kernel is accessing at
> > > do_translation_fault+0x30 is causing this to go recursive...
> > >
> > > I can reproduce this, pretty easily, with a similar enough trace, though
> > > faddr2line isn't happy to give me a line number.
> >
> > Did you reproduce it the same way with a few find /?
> >
> > I am trying to reproduce it and failing :/
> > (Using Torvald's HEAD)
> >
> > This is my config:
> >
> > CONFIG_HAVE_ARCH_KASAN=y
> > CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
> > CONFIG_CC_HAS_KASAN_GENERIC=y
> > CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
> > CONFIG_KASAN=y
> > CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
> > CONFIG_KASAN_GENERIC=y
> > CONFIG_KASAN_OUTLINE=y
> > # CONFIG_KASAN_INLINE is not set
> > # CONFIG_KASAN_STACK is not set
> > # CONFIG_KASAN_VMALLOC is not set
> > # CONFIG_KASAN_EXTRA_INFO is not set
> >
> > Do you use more KASAN?
>
> I used a config per Clement's instructions, i.e.
>
> | [mark@lakrids:~/src/linux]% git checkout v6.12-rc3
> | HEAD is now at 8e929cb546ee4 Linux 6.12-rc3
> | [mark@lakrids:~/src/linux]% git clean -qfdx
> | [mark@lakrids:~/src/linux]% echo 'CONFIG_KASAN=y' > arch/arm/configs/fragment-kasan.config
> | [mark@lakrids:~/src/linux]% usekorg 14.2.0 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig fragment-kasan.config
> | HOSTCC scripts/basic/fixdep
> | HOSTCC scripts/kconfig/conf.o
> | HOSTCC scripts/kconfig/confdata.o
> | HOSTCC scripts/kconfig/expr.o
> | LEX scripts/kconfig/lexer.lex.c
> | YACC scripts/kconfig/parser.tab.[ch]
> | HOSTCC scripts/kconfig/lexer.lex.o
> | HOSTCC scripts/kconfig/menu.o
> | HOSTCC scripts/kconfig/parser.tab.o
> | HOSTCC scripts/kconfig/preprocess.o
> | HOSTCC scripts/kconfig/symbol.o
> | HOSTCC scripts/kconfig/util.o
> | HOSTLD scripts/kconfig/conf
> | #
> | # configuration written to .config
> | #
> | Using .config as base
> | Merging ./arch/arm/configs/fragment-kasan.config
> | Value of CONFIG_KASAN is redefined by fragment ./arch/arm/configs/fragment-kasan.config:
> | Previous value: # CONFIG_KASAN is not set
> | New value: CONFIG_KASAN=y
> |
> | #
> | # merged configuration written to .config (needs make)
> | #
> | #
> | # configuration written to .config
> | #
> | [mark@lakrids:~/src/linux]% grep KASAN .config
> | CONFIG_KASAN_SHADOW_OFFSET=0x5f000000
> | CONFIG_HAVE_ARCH_KASAN=y
> | CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
> | CONFIG_CC_HAS_KASAN_GENERIC=y
> | CONFIG_KASAN=y
> | CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
> | CONFIG_KASAN_GENERIC=y
> | # CONFIG_KASAN_OUTLINE is not set
> | CONFIG_KASAN_INLINE=y
> | CONFIG_KASAN_STACK=y
> | CONFIG_KASAN_VMALLOC=y
> | # CONFIG_KASAN_MODULE_TEST is not set
> | # CONFIG_KASAN_EXTRA_INFO is not set
>
> IIUC you'll need CONFIG_KASAN_INLINE=y, CONFIG_KASAN_STACK=y, and
> CONFIG_KASAN_VMALLOC=y.
>
> With that, just booting and tapping a few keys was sufficient to trigger
> a crash using Debian the initrd Clement linked.
>
> [...]
>
> > > The relevant asm is:
> > (...)
> > > ... so we're using the new task's mm, but still executing in the context of the
> > > old task (and using its stack). I suspect the new task's mm doesn't have the
> > > old task's stack shadow mapped in, and AFAICT we don't map that in explicitly
> > > anywhere before we switch to the new mm.
> > >
> > > Linus, can you look into that?
> >
> > Yeah it looks like a spot-on identification of the problem, I can try to
> > think about how we could fix this if I can reproduce it, I keep trying
> > to provoke the crash :/
>
> It's a bit grotty -- AFAICT you'd either need to prefault in the
> specific part of the vmalloc space when switching tasks, or we'd need to
> preallocate all the shared vmalloc tables at the start of time so that
> they're always up-to-date.
>
> While we could disable KASAN_STACK, that's only going to mask the
> problem until this happens for any other vmalloc shadow...
>
Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
When I implemented VMAP_STACK for ARM, I added an explicit load from
the new stack while still running from the old one (in __switch_to) so
that the ordinary faulting code can deal with it. Couldn't we do the
same for the vmalloc shadow of the new stack?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:22 ` Ard Biesheuvel
@ 2024-10-15 14:35 ` Mark Rutland
2024-10-15 14:44 ` Ard Biesheuvel
0 siblings, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-15 14:35 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 04:22:20PM +0200, Ard Biesheuvel wrote:
> On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> > >
> > > > I think what's happening here is that when switching from prev to next
> > > > in the scheduler, we switch to next's mm before we actually switch to
> > > > next's register state, and there's a transient window where prev is
> > > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > > fault on this until we switch to the overflow stack.
[...]
> > > Yeah it looks like a spot-on identification of the problem, I can try to
> > > think about how we could fix this if I can reproduce it, I keep trying
> > > to provoke the crash :/
> >
> > It's a bit grotty -- AFAICT you'd either need to prefault in the
> > specific part of the vmalloc space when switching tasks, or we'd need to
> > preallocate all the shared vmalloc tables at the start of time so that
> > they're always up-to-date.
> >
> > While we could disable KASAN_STACK, that's only going to mask the
> > problem until this happens for any other vmalloc shadow...
>
> Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
It depends on what the vmalloc memory is used for; if it's anything else
used in the fault handling path, that'll fault recursively, and it's
possible that'll happen indirectly via other instrumentation.
> When I implemented VMAP_STACK for ARM, I added an explicit load from
> the new stack while still running from the old one (in __switch_to) so
> that the ordinary faulting code can deal with it. Couldn't we do the
> same for the vmalloc shadow of the new stack?
We could do something similar, but note that it's backwards: we need to
ensure that the old/current stack shadow will be mapped in the new mm.
So the usual fault handling can't handle that as-is, because you need to
fault-in pages for an mm which isn't yet in use. That logic could be
factored out and shared, though.
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:00 ` Mark Rutland
2024-10-15 14:22 ` Ard Biesheuvel
@ 2024-10-15 14:40 ` Linus Walleij
1 sibling, 0 replies; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 14:40 UTC (permalink / raw)
To: Mark Rutland
Cc: Clement LE GOFFIC, Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, Ard Biesheuvel, linux-stm32, Antonio Borneo
On Tue, Oct 15, 2024 at 4:00 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > I am trying to reproduce it and failing :/
> > (Using Torvald's HEAD)
(...)
> I used a config per Clement's instructions, i.e.
>
> | [mark@lakrids:~/src/linux]% git checkout v6.12-rc3
> | HEAD is now at 8e929cb546ee4 Linux 6.12-rc3
> | [mark@lakrids:~/src/linux]% git clean -qfdx
> | [mark@lakrids:~/src/linux]% echo 'CONFIG_KASAN=y' > arch/arm/configs/fragment-kasan.config
> | [mark@lakrids:~/src/linux]% usekorg 14.2.0 make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- vexpress_defconfig fragment-kasan.config
OK I got the crash, now also with my build scripts.
Dunno what I did wrong...
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:35 ` Mark Rutland
@ 2024-10-15 14:44 ` Ard Biesheuvel
2024-10-15 14:55 ` Linus Walleij
2024-10-15 15:26 ` Mark Rutland
0 siblings, 2 replies; 21+ messages in thread
From: Ard Biesheuvel @ 2024-10-15 14:44 UTC (permalink / raw)
To: Mark Rutland
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, 15 Oct 2024 at 16:35, Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Tue, Oct 15, 2024 at 04:22:20PM +0200, Ard Biesheuvel wrote:
> > On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
> > >
> > > On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > > > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> > > >
> > > > > I think what's happening here is that when switching from prev to next
> > > > > in the scheduler, we switch to next's mm before we actually switch to
> > > > > next's register state, and there's a transient window where prev is
> > > > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > > > fault on this until we switch to the overflow stack.
>
> [...]
>
> > > > Yeah it looks like a spot-on identification of the problem, I can try to
> > > > think about how we could fix this if I can reproduce it, I keep trying
> > > > to provoke the crash :/
> > >
> > > It's a bit grotty -- AFAICT you'd either need to prefault in the
> > > specific part of the vmalloc space when switching tasks, or we'd need to
> > > preallocate all the shared vmalloc tables at the start of time so that
> > > they're always up-to-date.
> > >
> > > While we could disable KASAN_STACK, that's only going to mask the
> > > problem until this happens for any other vmalloc shadow...
> >
> > Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
>
> It depends on what the vmalloc memory is used for; if it's anything else
> used in the fault handling path, that'll fault recursively, and it's
> possible that'll happen indirectly via other instrumentation.
>
> > When I implemented VMAP_STACK for ARM, I added an explicit load from
> > the new stack while still running from the old one (in __switch_to) so
> > that the ordinary faulting code can deal with it. Couldn't we do the
> > same for the vmalloc shadow of the new stack?
>
> We could do something similar, but note that it's backwards: we need to
> ensure that the old/current stack shadow will be mapped in the new mm.
>
> So the usual fault handling can't handle that as-is, because you need to
> fault-in pages for an mm which isn't yet in use. That logic could be
> factored out and shared, though.
>
Not sure I follow you here. The crash is in the kernel, no?
So there is only a single vmalloc space where all the mappings should
reside, but each process has its own copy of the top level page table,
which needs to be synced up when it goes stale.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:44 ` Ard Biesheuvel
@ 2024-10-15 14:55 ` Linus Walleij
2024-10-15 15:26 ` Mark Rutland
1 sibling, 0 replies; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 14:55 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Mark Rutland, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 4:45 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> On Tue, 15 Oct 2024 at 16:35, Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > On Tue, Oct 15, 2024 at 04:22:20PM +0200, Ard Biesheuvel wrote:
> > > On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
> > > >
> > > > On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > > > > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > > > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> > > > >
> > > > > > I think what's happening here is that when switching from prev to next
> > > > > > in the scheduler, we switch to next's mm before we actually switch to
> > > > > > next's register state, and there's a transient window where prev is
> > > > > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > > > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > > > > fault on this until we switch to the overflow stack.
> >
> > [...]
> >
> > > > > Yeah it looks like a spot-on identification of the problem, I can try to
> > > > > think about how we could fix this if I can reproduce it, I keep trying
> > > > > to provoke the crash :/
> > > >
> > > > It's a bit grotty -- AFAICT you'd either need to prefault in the
> > > > specific part of the vmalloc space when switching tasks, or we'd need to
> > > > preallocate all the shared vmalloc tables at the start of time so that
> > > > they're always up-to-date.
> > > >
> > > > While we could disable KASAN_STACK, that's only going to mask the
> > > > problem until this happens for any other vmalloc shadow...
> > >
> > > Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
> >
> > It depends on what the vmalloc memory is used for; if it's anything else
> > used in the fault handling path, that'll fault recursively, and it's
> > possible that'll happen indirectly via other instrumentation.
> >
> > > When I implemented VMAP_STACK for ARM, I added an explicit load from
> > > the new stack while still running from the old one (in __switch_to) so
> > > that the ordinary faulting code can deal with it. Couldn't we do the
> > > same for the vmalloc shadow of the new stack?
> >
> > We could do something similar, but note that it's backwards: we need to
> > ensure that the old/current stack shadow will be mapped in the new mm.
> >
> > So the usual fault handling can't handle that as-is, because you need to
> > fault-in pages for an mm which isn't yet in use. That logic could be
> > factored out and shared, though.
> >
>
> Not sure I follow you here. The crash is in the kernel, no?
>
> So there is only a single vmalloc space where all the mappings should
> reside, but each process has its own copy of the top level page table,
> which needs to be synced up when it goes stale.
That's how it works AFAICT.
The vmalloc/kernel space is mapped using the very same
actual page tables in all per-process MM contexts, so I also
think it's just a matter of syncing the top-level page table (PGD),
so I will try to do that.
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 14:44 ` Ard Biesheuvel
2024-10-15 14:55 ` Linus Walleij
@ 2024-10-15 15:26 ` Mark Rutland
2024-10-15 16:07 ` Ard Biesheuvel
1 sibling, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-15 15:26 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 04:44:56PM +0200, Ard Biesheuvel wrote:
> On Tue, 15 Oct 2024 at 16:35, Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > On Tue, Oct 15, 2024 at 04:22:20PM +0200, Ard Biesheuvel wrote:
> > > On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
> > > >
> > > > On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > > > > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > > > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> > > > >
> > > > > > I think what's happening here is that when switching from prev to next
> > > > > > in the scheduler, we switch to next's mm before we actually switch to
> > > > > > next's register state, and there's a transient window where prev is
> > > > > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > > > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > > > > fault on this until we switch to the overflow stack.
> >
> > [...]
> >
> > > > > Yeah it looks like a spot-on identification of the problem, I can try to
> > > > > think about how we could fix this if I can reproduce it, I keep trying
> > > > > to provoke the crash :/
> > > >
> > > > It's a bit grotty -- AFAICT you'd either need to prefault in the
> > > > specific part of the vmalloc space when switching tasks, or we'd need to
> > > > preallocate all the shared vmalloc tables at the start of time so that
> > > > they're always up-to-date.
> > > >
> > > > While we could disable KASAN_STACK, that's only going to mask the
> > > > problem until this happens for any other vmalloc shadow...
> > >
> > > Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
> >
> > It depends on what the vmalloc memory is used for; if it's anything else
> > used in the fault handling path, that'll fault recursively, and it's
> > possible that'll happen indirectly via other instrumentation.
> >
> > > When I implemented VMAP_STACK for ARM, I added an explicit load from
> > > the new stack while still running from the old one (in __switch_to) so
> > > that the ordinary faulting code can deal with it. Couldn't we do the
> > > same for the vmalloc shadow of the new stack?
> >
> > We could do something similar, but note that it's backwards: we need to
> > ensure that the old/current stack shadow will be mapped in the new mm.
> >
> > So the usual fault handling can't handle that as-is, because you need to
> > fault-in pages for an mm which isn't yet in use. That logic could be
> > factored out and shared, though.
>
> Not sure I follow you here. The crash is in the kernel, no?
Yep; I'm referring to the vmalloc space being lazily faulted-in and
copied from init_mm into the active pgd under do_translation_fault().
Looking some more, I don't see how VMAP_STACK guarantees that the
old/active stack is mapped in the new mm when switching from the old mm
to the new mm (which happens before __switch_to()).
Either I'm missing something, or we have a latent bug. Maybe we have
some explicit copying/prefaulting elsewhere I'm missing?
What happens when switching between two tasks whose stacks happen to be
in distinct sub-trees of the vmalloc tables?
> So there is only a single vmalloc space where all the mappings should
> reside, but each process has its own copy of the top level page table,
> which needs to be synced up when it goes stale.
Yep -- the problem is when we can safely do that syncing up, since the
lazy syncing in do_translation_fault() can't safely be used to sync
anything that's used during do_translation_fault(), including the stack,
etc.
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 15:26 ` Mark Rutland
@ 2024-10-15 16:07 ` Ard Biesheuvel
2024-10-15 16:27 ` Mark Rutland
0 siblings, 1 reply; 21+ messages in thread
From: Ard Biesheuvel @ 2024-10-15 16:07 UTC (permalink / raw)
To: Mark Rutland
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, 15 Oct 2024 at 17:26, Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Tue, Oct 15, 2024 at 04:44:56PM +0200, Ard Biesheuvel wrote:
> > On Tue, 15 Oct 2024 at 16:35, Mark Rutland <mark.rutland@arm.com> wrote:
> > >
> > > On Tue, Oct 15, 2024 at 04:22:20PM +0200, Ard Biesheuvel wrote:
> > > > On Tue, 15 Oct 2024 at 16:00, Mark Rutland <mark.rutland@arm.com> wrote:
> > > > >
> > > > > On Tue, Oct 15, 2024 at 03:51:02PM +0200, Linus Walleij wrote:
> > > > > > On Tue, Oct 15, 2024 at 12:28 PM Mark Rutland <mark.rutland@arm.com> wrote:
> > > > > > > On Mon, Oct 14, 2024 at 03:19:49PM +0200, Clement LE GOFFIC wrote:
> > > > > >
> > > > > > > I think what's happening here is that when switching from prev to next
> > > > > > > in the scheduler, we switch to next's mm before we actually switch to
> > > > > > > next's register state, and there's a transient window where prev is
> > > > > > > executed using next's mm. AFAICT we don't map prev's KASAN stack shadow
> > > > > > > into next's mm anywhere, and so inlined KASAN_STACK checks recursively
> > > > > > > fault on this until we switch to the overflow stack.
> > >
> > > [...]
> > >
> > > > > > Yeah it looks like a spot-on identification of the problem, I can try to
> > > > > > think about how we could fix this if I can reproduce it, I keep trying
> > > > > > to provoke the crash :/
> > > > >
> > > > > It's a bit grotty -- AFAICT you'd either need to prefault in the
> > > > > specific part of the vmalloc space when switching tasks, or we'd need to
> > > > > preallocate all the shared vmalloc tables at the start of time so that
> > > > > they're always up-to-date.
> > > > >
> > > > > While we could disable KASAN_STACK, that's only going to mask the
> > > > > problem until this happens for any other vmalloc shadow...
> > > >
> > > > Is the other vmalloc shadow not covered by the ordinary on-demand faulting?
> > >
> > > It depends on what the vmalloc memory is used for; if it's anything else
> > > used in the fault handling path, that'll fault recursively, and it's
> > > possible that'll happen indirectly via other instrumentation.
> > >
> > > > When I implemented VMAP_STACK for ARM, I added an explicit load from
> > > > the new stack while still running from the old one (in __switch_to) so
> > > > that the ordinary faulting code can deal with it. Couldn't we do the
> > > > same for the vmalloc shadow of the new stack?
> > >
> > > We could do something similar, but note that it's backwards: we need to
> > > ensure that the old/current stack shadow will be mapped in the new mm.
> > >
> > > So the usual fault handling can't handle that as-is, because you need to
> > > fault-in pages for an mm which isn't yet in use. That logic could be
> > > factored out and shared, though.
> >
> > Not sure I follow you here. The crash is in the kernel, no?
>
> Yep; I'm referring to the vmalloc space being lazily faulted-in and
> copied from init_mm into the active pgd under do_translation_fault().
>
> Looking some more, I don't see how VMAP_STACK guarantees that the
> old/active stack is mapped in the new mm when switching from the old mm
> to the new mm (which happens before __switch_to()).
>
> Either I'm missing something, or we have a latent bug. Maybe we have
> some explicit copying/prefaulting elsewhere I'm missing?
>
We bump the vmalloc_seq counter for that. Given that the top-level
page table can only gain entries covering the kernel space, this
should be sufficient for the old task's stack to be mapped in the new
task's page tables.
> What happens when switching between two tasks whose stacks happen to be
> in distinct sub-trees of the vmalloc tables?
>
> > So there is only a single vmalloc space where all the mappings should
> > reside, but each process has its own copy of the top level page table,
> > which needs to be synced up when it goes stale.
>
> Yep -- the problem is when we can safely do that syncing up, since the
> lazy syncing in do_translation_fault() can't safely be used to sync
> anything that's used during do_translation_fault(), including the stack,
> etc.
>
Indeed.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 16:07 ` Ard Biesheuvel
@ 2024-10-15 16:27 ` Mark Rutland
2024-10-15 17:28 ` Ard Biesheuvel
0 siblings, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-15 16:27 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 06:07:00PM +0200, Ard Biesheuvel wrote:
> On Tue, 15 Oct 2024 at 17:26, Mark Rutland <mark.rutland@arm.com> wrote:
> > Looking some more, I don't see how VMAP_STACK guarantees that the
> > old/active stack is mapped in the new mm when switching from the old mm
> > to the new mm (which happens before __switch_to()).
> >
> > Either I'm missing something, or we have a latent bug. Maybe we have
> > some explicit copying/prefaulting elsewhere I'm missing?
>
> We bump the vmalloc_seq counter for that. Given that the top-level
> page table can only gain entries covering the kernel space, this
> should be sufficient for the old task's stack to be mapped in the new
> task's page tables.
Ah, yep -- I had missed that. Thanks for the pointer!
From a superficial look, it sounds like it should be possible to extend
that to also handle the KASAN shadow of the vmalloc area (which
__check_vmalloc_seq() currently doesn't copy), but I'm not sure of
exactly when we initialise the shadow for a vmalloc allocation relative
to updating vmalloc_seq.
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 16:27 ` Mark Rutland
@ 2024-10-15 17:28 ` Ard Biesheuvel
2024-10-15 20:55 ` Linus Walleij
2024-10-16 8:55 ` Mark Rutland
0 siblings, 2 replies; 21+ messages in thread
From: Ard Biesheuvel @ 2024-10-15 17:28 UTC (permalink / raw)
To: Mark Rutland
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, 15 Oct 2024 at 18:27, Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Tue, Oct 15, 2024 at 06:07:00PM +0200, Ard Biesheuvel wrote:
> > On Tue, 15 Oct 2024 at 17:26, Mark Rutland <mark.rutland@arm.com> wrote:
> > > Looking some more, I don't see how VMAP_STACK guarantees that the
> > > old/active stack is mapped in the new mm when switching from the old mm
> > > to the new mm (which happens before __switch_to()).
> > >
> > > Either I'm missing something, or we have a latent bug. Maybe we have
> > > some explicit copying/prefaulting elsewhere I'm missing?
> >
> > We bump the vmalloc_seq counter for that. Given that the top-level
> > page table can only gain entries covering the kernel space, this
> > should be sufficient for the old task's stack to be mapped in the new
> > task's page tables.
>
> Ah, yep -- I had missed that. Thanks for the pointer!
>
> From a superficial look, it sounds like it should be possible to extend
> that to also handle the KASAN shadow of the vmalloc area (which
> __check_vmalloc_seq() currently doesn't copy), but I'm not sure of
> exactly when we initialise the shadow for a vmalloc allocation relative
> to updating vmalloc_seq.
>
Indeed. It appears both vmalloc_seq() and arch_sync_kernel_mappings()
need to take the vmalloc shadow into account specifically. And we may
also need the dummy read from the stack's shadow in __switch_to - I am
pretty sure I added that for a reason.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 17:28 ` Ard Biesheuvel
@ 2024-10-15 20:55 ` Linus Walleij
2024-10-15 21:24 ` Linus Walleij
2024-10-16 8:55 ` Mark Rutland
1 sibling, 1 reply; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 20:55 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Mark Rutland, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 7:28 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > From a superficial look, it sounds like it should be possible to extend
> > that to also handle the KASAN shadow of the vmalloc area (which
> > __check_vmalloc_seq() currently doesn't copy), but I'm not sure of
> > exactly when we initialise the shadow for a vmalloc allocation relative
> > to updating vmalloc_seq.
>
> Indeed. It appears both vmalloc_seq() and arch_sync_kernel_mappings()
> need to take the vmalloc shadow into account specifically.
I'm trying to look into that.
> And we may
> also need the dummy read from the stack's shadow in __switch_to - I am
> pretty sure I added that for a reason.
I added that since it was simple:
From c3c76df2a9b8132b169809dcdf93488cb43a2688 Mon Sep 17 00:00:00 2001
From: Linus Walleij <linus.walleij@linaro.org>
Date: Tue, 15 Oct 2024 22:50:38 +0200
Subject: [PATCH] ARM: entry: Do a dummy read from VMAP shadow
When switching task, in addition to a dummy read from the new
VMAP stack, also do a dummy read from the VMAP stack's
corresponding KASAN shadow memory to sync things up in
the new MM context.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
---
arch/arm/kernel/entry-armv.S | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index 1dfae1af8e31..ed2d0d89e2e9 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -25,6 +25,7 @@
#include <asm/tls.h>
#include <asm/system_info.h>
#include <asm/uaccess-asm.h>
+#include <asm/kasan_def.h>
#include "entry-header.S"
#include <asm/probes.h>
@@ -561,6 +562,13 @@ ENTRY(__switch_to)
@ entries covering the vmalloc region.
@
ldr r2, [ip]
+#ifdef CONFIG_KASAN
+ @ Also dummy read from the KASAN shadow memory for the new stack if we
+ @ are using KASAN
+ mov_l r2, KASAN_SHADOW_OFFSET
+ add r2, ip, lsr #KASAN_SHADOW_SCALE_SHIFT
+ ldr r2, [r2]
+#endif
#endif
@ When CONFIG_THREAD_INFO_IN_TASK=n, the update of SP itself is what
--
2.46.2
We still get crashes in do_translation_fault() so we need the more
thorough solution. Maybe this is needed too.
Yours,
Linus Walleij
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 20:55 ` Linus Walleij
@ 2024-10-15 21:24 ` Linus Walleij
0 siblings, 0 replies; 21+ messages in thread
From: Linus Walleij @ 2024-10-15 21:24 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Mark Rutland, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 10:55 PM Linus Walleij <linus.walleij@linaro.org> wrote:
> On Tue, Oct 15, 2024 at 7:28 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> > > From a superficial look, it sounds like it should be possible to extend
> > > that to also handle the KASAN shadow of the vmalloc area (which
> > > __check_vmalloc_seq() currently doesn't copy), but I'm not sure of
> > > exactly when we initialise the shadow for a vmalloc allocation relative
> > > to updating vmalloc_seq.
> >
> > Indeed. It appears both vmalloc_seq() and arch_sync_kernel_mappings()
> > need to take the vmalloc shadow into account specifically.
>
> I'm trying to look into that.
I fixed that too and now the KASAN is stabilized. I'll send out the
patches so we get something to test.
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-15 17:28 ` Ard Biesheuvel
2024-10-15 20:55 ` Linus Walleij
@ 2024-10-16 8:55 ` Mark Rutland
2024-10-16 19:00 ` Linus Walleij
1 sibling, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-16 8:55 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Tue, Oct 15, 2024 at 07:28:06PM +0200, Ard Biesheuvel wrote:
> On Tue, 15 Oct 2024 at 18:27, Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > On Tue, Oct 15, 2024 at 06:07:00PM +0200, Ard Biesheuvel wrote:
> > > On Tue, 15 Oct 2024 at 17:26, Mark Rutland <mark.rutland@arm.com> wrote:
> > > > Looking some more, I don't see how VMAP_STACK guarantees that the
> > > > old/active stack is mapped in the new mm when switching from the old mm
> > > > to the new mm (which happens before __switch_to()).
> > > >
> > > > Either I'm missing something, or we have a latent bug. Maybe we have
> > > > some explicit copying/prefaulting elsewhere I'm missing?
> > >
> > > We bump the vmalloc_seq counter for that. Given that the top-level
> > > page table can only gain entries covering the kernel space, this
> > > should be sufficient for the old task's stack to be mapped in the new
> > > task's page tables.
> >
> > Ah, yep -- I had missed that. Thanks for the pointer!
> >
> > From a superficial look, it sounds like it should be possible to extend
> > that to also handle the KASAN shadow of the vmalloc area (which
> > __check_vmalloc_seq() currently doesn't copy), but I'm not sure of
> > exactly when we initialise the shadow for a vmalloc allocation relative
> > to updating vmalloc_seq.
> >
>
> Indeed. It appears both vmalloc_seq() and arch_sync_kernel_mappings()
> need to take the vmalloc shadow into account specifically. And we may
> also need the dummy read from the stack's shadow in __switch_to - I am
> pretty sure I added that for a reason.
I believe that's necessary for the lazy TLB switch, at least for SMP:
// CPU 0 // CPU 1
<< switches to task X's mm >>
<< creates kthread task Y >>
<< maps task Y's new stack >>
<< maps task Y's new shadow >>
// Y switched out
context_switch(..., Y, ..., ...);
// Switch from X to Y
context_switch(..., X, Y, ...) {
// prev = X
// next = Y
if (!next->mm) {
// Y has no mm
// No switch_mm() here
// ... so no check_vmalloc_seq()
} else {
// not taken
}
...
// X's mm still lacks Y's stack + shadow here
switch_to(prev, next, prev);
}
... so probably worth a comment that we're faulting in the new
stack+shadow for for lazy tlb when switching to a task with no mm?
In the lazy tlb case the current/old mappings don't disappear from the
active mm, and so we don't need to go add those to the new mm, which is what
we need check_vmalloc_seq() for.
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-16 8:55 ` Mark Rutland
@ 2024-10-16 19:00 ` Linus Walleij
2024-10-17 10:09 ` Mark Rutland
0 siblings, 1 reply; 21+ messages in thread
From: Linus Walleij @ 2024-10-16 19:00 UTC (permalink / raw)
To: Mark Rutland
Cc: Ard Biesheuvel, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Wed, Oct 16, 2024 at 10:55 AM Mark Rutland <mark.rutland@arm.com> wrote:
> I believe that's necessary for the lazy TLB switch, at least for SMP:
>
> // CPU 0 // CPU 1
>
> << switches to task X's mm >>
>
> << creates kthread task Y >>
> << maps task Y's new stack >>
> << maps task Y's new shadow >>
>
> // Y switched out
> context_switch(..., Y, ..., ...);
>
> // Switch from X to Y
> context_switch(..., X, Y, ...) {
> // prev = X
> // next = Y
>
> if (!next->mm) {
> // Y has no mm
> // No switch_mm() here
> // ... so no check_vmalloc_seq()
> } else {
> // not taken
> }
>
> ...
>
> // X's mm still lacks Y's stack + shadow here
>
> switch_to(prev, next, prev);
> }
>
> ... so probably worth a comment that we're faulting in the new
> stack+shadow for for lazy tlb when switching to a task with no mm?
Switching to a task with no mm == switching to a kernel daemon.
And those only use the kernel memory and relies on that always
being mapped in any previous mm context, right.
But where do we put that comment? In kernel/sched/core.c
context_switch()?
It's no different in any architecture I think, and they pretty much all
use KASAN these days.
Or in ARM32's enter_lazy_tlb() in arch/arm/include/asm/mmu_context.h?
I'm unsure. I would make it a separate patch.
> In the lazy tlb case the current/old mappings don't disappear from the
> active mm, and so we don't need to go add those to the new mm, which is what
> we need check_vmalloc_seq() for.
Yups that's how I understand it too.
Yours,
Linus Walleij
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-16 19:00 ` Linus Walleij
@ 2024-10-17 10:09 ` Mark Rutland
2024-10-17 10:31 ` Ard Biesheuvel
0 siblings, 1 reply; 21+ messages in thread
From: Mark Rutland @ 2024-10-17 10:09 UTC (permalink / raw)
To: Linus Walleij, Ard Biesheuvel
Cc: Clement LE GOFFIC, Russell King, Russell King (Oracle), Kees Cook,
AngeloGioacchino Del Regno, Mark Brown, linux-arm-kernel,
linux-kernel, linux-stm32, Antonio Borneo
On Wed, Oct 16, 2024 at 09:00:22PM +0200, Linus Walleij wrote:
> On Wed, Oct 16, 2024 at 10:55 AM Mark Rutland <mark.rutland@arm.com> wrote:
>
> > I believe that's necessary for the lazy TLB switch, at least for SMP:
> >
> > // CPU 0 // CPU 1
> >
> > << switches to task X's mm >>
> >
> > << creates kthread task Y >>
> > << maps task Y's new stack >>
> > << maps task Y's new shadow >>
> >
> > // Y switched out
> > context_switch(..., Y, ..., ...);
> >
> > // Switch from X to Y
> > context_switch(..., X, Y, ...) {
> > // prev = X
> > // next = Y
> >
> > if (!next->mm) {
> > // Y has no mm
> > // No switch_mm() here
> > // ... so no check_vmalloc_seq()
> > } else {
> > // not taken
> > }
> >
> > ...
> >
> > // X's mm still lacks Y's stack + shadow here
> >
> > switch_to(prev, next, prev);
> > }
> >
> > ... so probably worth a comment that we're faulting in the new
> > stack+shadow for for lazy tlb when switching to a task with no mm?
>
> Switching to a task with no mm == switching to a kernel daemon.
A common misconception, but not always true:
* A kernel thread can have an mm: see kthread_use_mm() and
kthread_unuse_mm().
* A user thread can lose its mm while exiting: see how do_exit() calls
exit_mm(), and how hte task remains preemptible for a while
thereafter.
... so we really do just mean "a task with no mm".
> And those only use the kernel memory and relies on that always
> being mapped in any previous mm context, right.
A task with no mm only uses kernel memory. Anything it uses must be
mapped in init_mm, but *might* not have been copied into every other mm,
and hence might not be in the previous mm context as per the example
above.
> But where do we put that comment? In kernel/sched/core.c
> context_switch()?
I was trying to suggest we update the existing comment in switch_to() to
be more explicit. e.g. expand the existing comment:
@
@ Do a dummy read from the new stack while running from the old one so
@ that we can rely on do_translation_fault() to fix up any stale PMD
@ entries covering the vmalloc region.
@
... with:
@
@ For a non-lazy mm switch, check_vmalloc_seq() has ensured that
@ that the active mm's page tables have mappings for the prev
@ task's stack and the next task's stack.
@
@ For a lazy mm switch the active mm's page tables have mappings
@ for the prev task's stack but might not have mappings for the
@ new taks stack. Do a dummy read from the new stack while
@ running from the old stack so that we can rely on
@ do_translation_fault() to fix up any stale PMD entries
@ covering the vmalloc region.
@
Ard, does that sound good to you?
Mark.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Crash on armv7-a using KASAN
2024-10-17 10:09 ` Mark Rutland
@ 2024-10-17 10:31 ` Ard Biesheuvel
0 siblings, 0 replies; 21+ messages in thread
From: Ard Biesheuvel @ 2024-10-17 10:31 UTC (permalink / raw)
To: Mark Rutland
Cc: Linus Walleij, Clement LE GOFFIC, Russell King,
Russell King (Oracle), Kees Cook, AngeloGioacchino Del Regno,
Mark Brown, linux-arm-kernel, linux-kernel, linux-stm32,
Antonio Borneo
On Thu, 17 Oct 2024 at 12:10, Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Wed, Oct 16, 2024 at 09:00:22PM +0200, Linus Walleij wrote:
> > On Wed, Oct 16, 2024 at 10:55 AM Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > > I believe that's necessary for the lazy TLB switch, at least for SMP:
> > >
> > > // CPU 0 // CPU 1
> > >
> > > << switches to task X's mm >>
> > >
> > > << creates kthread task Y >>
> > > << maps task Y's new stack >>
> > > << maps task Y's new shadow >>
> > >
> > > // Y switched out
> > > context_switch(..., Y, ..., ...);
> > >
> > > // Switch from X to Y
> > > context_switch(..., X, Y, ...) {
> > > // prev = X
> > > // next = Y
> > >
> > > if (!next->mm) {
> > > // Y has no mm
> > > // No switch_mm() here
> > > // ... so no check_vmalloc_seq()
> > > } else {
> > > // not taken
> > > }
> > >
> > > ...
> > >
> > > // X's mm still lacks Y's stack + shadow here
> > >
> > > switch_to(prev, next, prev);
> > > }
> > >
> > > ... so probably worth a comment that we're faulting in the new
> > > stack+shadow for for lazy tlb when switching to a task with no mm?
> >
> > Switching to a task with no mm == switching to a kernel daemon.
>
> A common misconception, but not always true:
>
> * A kernel thread can have an mm: see kthread_use_mm() and
> kthread_unuse_mm().
>
> * A user thread can lose its mm while exiting: see how do_exit() calls
> exit_mm(), and how hte task remains preemptible for a while
> thereafter.
>
> ... so we really do just mean "a task with no mm".
>
> > And those only use the kernel memory and relies on that always
> > being mapped in any previous mm context, right.
>
> A task with no mm only uses kernel memory. Anything it uses must be
> mapped in init_mm, but *might* not have been copied into every other mm,
> and hence might not be in the previous mm context as per the example
> above.
>
> > But where do we put that comment? In kernel/sched/core.c
> > context_switch()?
>
> I was trying to suggest we update the existing comment in switch_to() to
> be more explicit. e.g. expand the existing comment:
>
> @
> @ Do a dummy read from the new stack while running from the old one so
> @ that we can rely on do_translation_fault() to fix up any stale PMD
> @ entries covering the vmalloc region.
> @
>
> ... with:
>
> @
> @ For a non-lazy mm switch, check_vmalloc_seq() has ensured that
> @ that the active mm's page tables have mappings for the prev
> @ task's stack and the next task's stack.
> @
> @ For a lazy mm switch the active mm's page tables have mappings
> @ for the prev task's stack but might not have mappings for the
> @ new taks stack. Do a dummy read from the new stack while
task
> @ running from the old stack so that we can rely on
> @ do_translation_fault() to fix up any stale PMD entries
> @ covering the vmalloc region.
Might as well be more precise here, and say "populate missing PMD
entries covering the new task's stack in the old task's page tables"
> @
>
> Ard, does that sound good to you?
>
Yes.
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2024-10-17 11:01 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-10-14 13:19 Crash on armv7-a using KASAN Clement LE GOFFIC
2024-10-15 7:55 ` Linus Walleij
2024-10-15 10:35 ` Clement LE GOFFIC
2024-10-15 10:28 ` Mark Rutland
2024-10-15 13:51 ` Linus Walleij
2024-10-15 14:00 ` Mark Rutland
2024-10-15 14:22 ` Ard Biesheuvel
2024-10-15 14:35 ` Mark Rutland
2024-10-15 14:44 ` Ard Biesheuvel
2024-10-15 14:55 ` Linus Walleij
2024-10-15 15:26 ` Mark Rutland
2024-10-15 16:07 ` Ard Biesheuvel
2024-10-15 16:27 ` Mark Rutland
2024-10-15 17:28 ` Ard Biesheuvel
2024-10-15 20:55 ` Linus Walleij
2024-10-15 21:24 ` Linus Walleij
2024-10-16 8:55 ` Mark Rutland
2024-10-16 19:00 ` Linus Walleij
2024-10-17 10:09 ` Mark Rutland
2024-10-17 10:31 ` Ard Biesheuvel
2024-10-15 14:40 ` Linus Walleij
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).