* [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb [not found] <trinity-cc5e013b-ac22-4e47-a902-57a6cac67965-1549350557264@3c-app-gmx-bs52> @ 2019-02-05 15:02 ` Frank Wunderlich 2019-02-05 18:27 ` Greg KH 0 siblings, 1 reply; 4+ messages in thread From: Frank Wunderlich @ 2019-02-05 15:02 UTC (permalink / raw) To: stable Hi, i've noticed panics on my bananapi-r2 with kernel 4.14.95, but can't locate them in detail...all is pointing to kmem_cache_alloc and build_skb, but over different ways. i currently don't know what is triggering this crash, so i cannot reproduce them on my test-machine :( it's not only an oops...the machine stays still and have to be resettet, sometimes after an half hour running i had kernel 4.14.78 running before, and now fallback to it, because it is working. so i think it's a change between these 2 Kernel-versions and no OS-related (debian-stretch) or Hardware-issue i looked through git-tree but did not found anything related regards Frank here my 3 recorded crash-logs...maybe you can point me in the right direction: [ 1281.724002] Unable to handle kernel NULL pointer dereference at virtual address 00000001 [ 1281.732105] pgd = c0004000 [ 1281.734836] [00000001] *pgd=00000000 [ 1281.738425] Internal error: Oops: 5 [#1] SMP ARM [ 1281.743016] Modules linked in: xt_addrtype veth pppoe pppox ppp_generic slhc sit tunnel4 ip_tunnel bridge ip6t_REJECT nf_reject_ipv6 xt_mac ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_limit xt_iprange xt_recent xt_conntrack iptabl e_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat tun xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrac k xt_mark iptable_mangle mt76x2e mt76 mtk_thermal spi_mt65xx thermal_sys mt6577_auxadc pwm_mediatek nvmem_mtk_efuse mtk_pmic_keys evbug rtc_mt6 397 nfsd ip_tables x_tables ipv6 [ 1281.795425] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.95-bpi-r2-main #49 [ 1281.802510] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1281.808043] task: c11073c0 task.stack: c1100000 [ 1281.812551] PC is at kmem_cache_alloc+0xd0/0x21c [ 1281.817134] LR is at 0x391d6 [ 1281.819990] pc : [<c028c0f4>] lr : [<000391d6>] psr: 280f0113 [ 1281.826209] sp : c1101c18 ip : a80f0113 fp : c1101c4c [ 1281.831393] r10: 00000001 r9 : dbdf6900 r8 : c0895b9c [ 1281.836578] r7 : de801cc0 r6 : 01080020 r5 : c1226790 r4 : de801cc0 [ 1281.843056] r3 : 00000000 r2 : df5a39d4 r1 : 1e4e9000 r0 : 000391d5 [ 1281.849535] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 1281.856617] Control: 10c5387d Table: 9bdc006a DAC: 00000051 [ 1281.862319] Process swapper/0 (pid: 0, stack limit = 0xc1100218) [ 1281.868280] Stack: (0xc1101c18 to 0xc1102000) [ 1281.872608] 1c00: db119900 df5a0fb8 [ 1281.880733] 1c20: c1101c54 df5a0784 00000000 00000300 da308b00 680f0113 00000000 da1dd6b0 [ 1281.888858] 1c40: c1101c74 c1101c50 c0895b9c c028c030 df5a0784 00000000 00000300 da308b00 [ 1281.896983] 1c60: 680f0113 00000000 c1101c9c c1101c78 c0895d10 c0895b70 00000000 dcc48b78 [ 1281.905109] 1c80: db6e5f00 dcc48580 de8ec460 c1101d10 c1101d04 c1101ca0 c0a729cc c0895c88 [ 1281.913234] 1ca0: c015ff18 c01593b4 d63e99d9 00000000 c1101d0c c01525cc c0b02a08 df5a0f80 [ 1281.921359] 1cc0: c1101ce4 c1101cd0 c01525cc c0151994 df5a0f80 da029e00 c1101d1c 00000000 [ 1281.929484] 1ce0: 00000000 de8ec460 bf16bb90 dcc48b78 00000001 00000003 c1101d3c c1101d08 [ 1281.937608] 1d00: c0a72ba8 c0a727b4 00000002 c04e15c8 00000000 00000000 c04e15c8 de8ecd20 [ 1281.945732] 1d20: dcc48b78 00000002 bf16bb90 de8ecd20 c1101d54 c1101d40 bf16bbd4 c0a72b74 [ 1281.953858] 1d40: dcc48580 de8ecab0 c1101d8c c1101d58 c0a78f20 bf16bb9c c1101d64 de8ec460 [ 1281.961983] 1d60: ff76abc0 de8ecd20 de8ed868 c1101db0 bf16baf8 c10af2a8 ffffe000 00000040 [ 1281.970108] 1d80: c1101d9c c1101d90 c0a78fbc c0a78e90 c1101e04 c1101da0 bf16c0a8 c0a78fac [ 1281.978232] 1da0: de8ecd20 00200000 c1101ddc de8ecd20 c1101db0 c1101db0 00000000 00000000 [ 1281.986357] 1dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1281.994482] 1de0: c0aa6524 de8ed864 de8ed868 00000000 00000000 c10af2a8 c1101e2c c1101e08 [ 1282.002607] 1e00: c012cf74 bf16c058 c1102098 00000006 c11bcd08 00000100 00000007 ffffe000 [ 1282.010732] 1e20: c1101e8c c1101e30 c0101614 c012cf10 c0182400 c0185260 00000001 00200102 [ 1282.018858] 1e40: c1102d00 00017f7c c0b023a8 0000000a c11f4c40 c1103c6c c10af310 c1102080 [ 1282.026982] 1e60: c0185740 c10b7b3c 00000000 00000000 00000001 de819000 e1003000 e07fcc00 [ 1282.035107] 1e80: c1101e9c c1101e90 c012c998 c01014f4 c1101ec4 c1101ea0 c01818f4 c012c8c0 [ 1282.043231] 1ea0: c11043f8 c1168010 e100200c e1002000 c1101ef0 e1003000 c1101eec c1101ec8 [ 1282.051357] 1ec0: c01014a0 c0181890 c01098b0 680f0013 ffffffff c1101f24 c0d2c0e0 c1100000 [ 1282.059481] 1ee0: c1101f4c c1101ef0 c010e24c c0101450 00000000 0018f100 1e4e9000 c011f8e0 [ 1282.067607] 1f00: ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 e07fcc00 c1101f4c [ 1282.075731] 1f20: c1101f50 c1101f40 c01098ac c01098b0 680f0013 ffffffff 00000051 00000000 [ 1282.083856] 1f40: c1101f5c c1101f50 c0ac2d88 c0109874 c1101f84 c1101f60 c016e3b0 c0ac2d64 [ 1282.091982] 1f60: 000000be c11f3f00 c1103c40 ffffffff c11f3f00 c1079e30 c1101f94 c1101f88 [ 1282.100107] 1f80: c016e6b0 c016e2d0 c1101fac c1101f98 c0abcc9c c016e694 c11f3f40 c11f3f00 [ 1282.108231] 1fa0: c1101ff4 c1101fb0 c1000e28 c0abcbe8 ffffffff ffffffff 00000000 c1000784 [ 1282.116357] 1fc0: 00000000 c1079e30 00000000 c11f4494 c1103c58 c1079e2c c1108800 8000406a [ 1282.124481] 1fe0: 410fc073 00000000 00000000 c1101ff8 8000807c c1000a7c 00000000 00000000 [ 1282.132624] [<c028c0f4>] (kmem_cache_alloc) from [<c0895b9c>] (__build_skb+0x38/0xa0) [ 1282.140411] [<c0895b9c>] (__build_skb) from [<c0895d10>] (__netdev_alloc_skb+0x94/0x110) [ 1282.148456] [<c0895d10>] (__netdev_alloc_skb) from [<c0a729cc>] (__ieee80211_beacon_get+0x224/0x3a0) [ 1282.157535] [<c0a729cc>] (__ieee80211_beacon_get) from [<c0a72ba8>] (ieee80211_beacon_get_tim+0x40/0x11c) [ 1282.167077] [<c0a72ba8>] (ieee80211_beacon_get_tim) from [<bf16bbd4>] (mt76x2_update_beacon_iter+0x44/0x5c [mt76x2e]) [ 1282.177647] [<bf16bbd4>] (mt76x2_update_beacon_iter [mt76x2e]) from [<c0a78f20>] (__iterate_interfaces+0x9c/0x11c) [ 1282.187936] [<c0a78f20>] (__iterate_interfaces) from [<c0a78fbc>] (ieee80211_iterate_active_interfaces_atomic+0x1c/0x20) [ 1282.198756] [<c0a78fbc>] (ieee80211_iterate_active_interfaces_atomic) from [<bf16c0a8>] (mt76x2_pre_tbtt_tasklet+0x5c/0x148 [mt76x2e]) [ 1282.210783] [<bf16c0a8>] (mt76x2_pre_tbtt_tasklet [mt76x2e]) from [<c012cf74>] (tasklet_action+0x70/0x104) [ 1282.220379] [<c012cf74>] (tasklet_action) from [<c0101614>] (__do_softirq+0x12c/0x378) [ 1282.228248] [<c0101614>] (__do_softirq) from [<c012c998>] (irq_exit+0xe4/0x15c) [ 1282.235516] [<c012c998>] (irq_exit) from [<c01818f4>] (__handle_domain_irq+0x70/0xc4) [ 1282.243299] [<c01818f4>] (__handle_domain_irq) from [<c01014a0>] (gic_handle_irq+0x5c/0xa0) [ 1282.251601] [<c01014a0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1282.259031] Exception stack(0xc1101ef0 to 0xc1101f38) [ 1282.264048] 1ee0: 00000000 0018f100 1e4e9000 c011f8e0 [ 1282.272173] 1f00: ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 e07fcc00 c1101f4c [ 1282.280296] 1f20: c1101f50 c1101f40 c01098ac c01098b0 680f0013 ffffffff [ 1282.286872] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 1282.294225] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 1282.302267] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 1282.309877] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 1282.317402] [<c016e6b0>] (cpu_startup_entry) from [<c0abcc9c>] (rest_init+0xc0/0xc4) [ 1282.325101] [<c0abcc9c>] (rest_init) from [<c1000e28>] (start_kernel+0x3b8/0x3c4) [ 1282.332539] Code: e121f00c e35a0000 0a000002 e5943014 (e79a3003) Message from[ 1282.338703] ---[ end trace 8329366c083f9bce ]--- syslogd@bpi-r2-[ 1282.344651] Kernel panic - not syncing: Fatal exception in interrupt [ 1282.352323] CPU2: stopping [ 1282.355026] CPU: 2 PID: 3188 Comm: dmesg Tainted: G D 4.14.95-bpi-r2-main #49 [ 1282.363232] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1282.368786] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1282.376485] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1282.383666] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1282.391017] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1282.398539] [<c01014e0>] (gic_handle_irq) from [<c010e614>] (__irq_usr+0x54/0x80) [ 1282.405970] Exception stack(0xd8149fb0 to 0xd8149ff8) [ 1282.410987] 9fa0: beaaa9e8 004c62e1 00000000 00000032 [ 1282.419113] 9fc0: beaaa9e4 004c62e4 beaaa980 004c62e0 b6f144c0 004cb758 beaaa9e0 beaaa9e4 [ 1282.427236] 9fe0: 004c60b4 beaaa8e8 beaaa9e4 b6de2a3c 800f0030 ffffffff [ 1282.433804] CPU1: stopping [ 1282.436496] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 4.14.95-bpi-r2-main #49 [ 1282.444789] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1282.450328] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1282.458024] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1282.465202] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1282.472552] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1282.480073] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1282.487503] Exception stack(0xde943f38 to 0xde943f80) [ 1282.492517] 3f20: 00000000 000b9204 [ 1282.500642] 3f40: 1e4fa000 c011f8e0 ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 [ 1282.508767] 3f60: 00000000 de943f94 de943f98 de943f88 c01098ac c01098b0 60000013 ffffffff [ 1282.516895] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 1282.524247] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 1282.532289] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 1282.539898] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 1282.547421] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194) [ 1282.556409] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec) [ 1282.563755] CPU3: stopping [ 1282.566447] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 4.14.95-bpi-r2-main #49 [ 1282.574739] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1282.580279] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1282.587975] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1282.595152] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1282.602503] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1282.610024] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1282.617454] Exception stack(0xde947ea0 to 0xde947ee8) [ 1282.622474] 7ea0: dbc09200 00000001 dbc09200 0000b59f c10b7f80 00000000 df5d3f80 1e51c000 [ 1282.630598] 7ec0: c0abdbfc ffffe000 de91c800 de947f34 de947f38 de947ef0 c0abdbfc c015028c [ 1282.638718] 7ee0: 600d0013 ffffffff [ 1282.642188] [<c010e24c>] (__irq_svc) from [<c015028c>] (finish_task_switch+0xf0/0x244) [ 1282.650059] [<c015028c>] (finish_task_switch) from [<c0abdbfc>] (__schedule+0x21c/0x6f8) [ 1282.658101] [<c0abdbfc>] (__schedule) from [<c0abe500>] (schedule_idle+0x44/0x84) [ 1282.665538] [<c0abe500>] (schedule_idle) from [<c016e3d4>] (do_idle+0x110/0x150) [ 1282.672889] [<c016e3d4>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 1282.680413] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194) [ 1282.689400] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec) [ 1282.696754] ---[ end Kernel panic - not syncing: Fatal exception in interrupt e at Feb 4 19:03:43 ... kernel:[ 1281.738425] Internal error: Oops: 5 [#1] SMP ARM ============================================================================================================================= 19:15] root@bpi-r2-e:~ (156)# [ 666.687166] Unable to handle kernel NULL pointer dereference at virtual address 00000001 [ 666.695237] pgd = c0004000 [ 666.697927] [00000001] *pgd=00000000 [ 666.701484] Internal error: Oops: 5 [#1] SMP ARM [ 666.706066] Modules linked in: xt_addrtype veth pppoe pppox ppp_generic slhc sit tunnel4 ip_tunnel bridge ip6t_REJECT nf_reject_ipv6 xt_mac ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_limit xt_iprange xt_recent xt_conntrack iptabl e_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 tun xt_nat xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrac k xt_mark iptable_mangle spi_mt65xx mtk_thermal thermal_sys evbug mt6577_auxadc pwm_mediatek mtk_pmic_keys rtc_mt6397 nvmem_mtk_efuse nfsd ip_t ables x_tables ipv6 [last unloaded: mt76] [ 666.759246] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.95-bpi-r2-main #49 [ 666.766329] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 666.771860] task: c11073c0 task.stack: c1100000 [ 666.776363] PC is at kmem_cache_alloc+0xd0/0x21c [ 666.780945] LR is at 0x179ab [ 666.783801] pc : [<c028c0f4>] lr : [<000179ab>] psr: 200f0113 [ 666.790020] sp : c1101cd0 ip : a00f0113 fp : c1101d04 [ 666.795204] r10: 00000001 r9 : de97cb40 r8 : c0895b9c [ 666.800389] r7 : de801cc0 r6 : 01080020 r5 : c1226790 r4 : de801cc0 [ 666.806867] r3 : 00000000 r2 : df5a39d4 r1 : 1e4e9000 r0 : 000179aa [ 666.813347] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 666.820429] Control: 10c5387d Table: 9bbc806a DAC: 00000051 [ 666.826133] Process swapper/0 (pid: 0, stack limit = 0xc1100218) [ 666.832093] Stack: (0xc1101cd0 to 0xc1102000) [ 666.836423] 1cc0: 9a7c0c40 df9b1700 00000500 da723ec0 [ 666.844549] 1ce0: 00000740 00000740 da723ec0 dd8a0000 9a7c0500 deb6c728 c1101d2c c1101d08 [ 666.852674] 1d00: c0895b9c c028c030 da723ec0 00000740 da723ec0 e1038b00 dd8a0000 9a7c0500 [ 666.860799] 1d20: c1101d44 c1101d30 c0895c24 c0895b70 deb6c628 000005b0 c1101dbc c1101d48 [ 666.868924] 1d40: c071b4a0 c0895c10 00000002 00000000 c0dba83c 00000040 deb6c010 c1225940 [ 666.877050] 1d60: 3fff0000 00000000 c0b0184c 9a723f00 e1033000 c5ee0004 c08a9a98 c08a6ef0 [ 666.885175] 1d80: 00000040 da7c04c0 2a678002 00000000 c1101dbc deb6c728 00000001 00000040 [ 666.893299] 1da0: c11c0048 0000012c 00008f3d c1101df0 c1101e2c c1101dc0 c08ae4d8 c071b328 [ 666.901424] 1dc0: dd889940 dead4d00 c1101de4 c0debe30 c1104fcc c1103c6c ffffe000 c11bcb49 [ 666.909550] 1de0: c10b8840 df5a1840 1e4e9000 c1102d00 c1101df0 c1101df0 c1101df8 c1101df8 [ 666.917675] 1e00: ffffe000 c110208c 00000003 c11bcd08 00000100 00000004 ffffe000 00000008 [ 666.925800] 1e20: c1101e8c c1101e30 c0101614 c08ae39c c0182400 c0185260 00000001 00200102 [ 666.933925] 1e40: c1102d00 00008f3d c0b023a8 0000000a c11f4c40 c1103c6c c10af310 c1102080 [ 666.942050] 1e60: c0185740 c10b7b3c 00000000 00000000 00000001 de819000 e1003000 e07fcc00 [ 666.950176] 1e80: c1101e9c c1101e90 c012c998 c01014f4 c1101ec4 c1101ea0 c01818f4 c012c8c0 [ 666.958302] 1ea0: c11043f8 c1168010 e100200c e1002000 c1101ef0 e1003000 c1101eec c1101ec8 [ 666.966428] 1ec0: c01014a0 c0181890 c01098b0 600f0013 ffffffff c1101f24 c0d2c0e0 c1100000 [ 666.974553] 1ee0: c1101f4c c1101ef0 c010e24c c0101450 00000000 000aeb14 1e4e9000 c011f8e0 [ 666.982678] 1f00: ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 e07fcc00 c1101f4c [ 666.990803] 1f20: c1101f50 c1101f40 c01098ac c01098b0 600f0013 ffffffff 00000051 00000000 [ 666.998927] 1f40: c1101f5c c1101f50 c0ac2d88 c0109874 c1101f84 c1101f60 c016e3b0 c0ac2d64 [ 667.007053] 1f60: 000000be c11f3f00 c1103c40 ffffffff c11f3f00 c1079e30 c1101f94 c1101f88 [ 667.015180] 1f80: c016e6b0 c016e2d0 c1101fac c1101f98 c0abcc9c c016e694 c11f3f40 c11f3f00 [ 667.023305] 1fa0: c1101ff4 c1101fb0 c1000e28 c0abcbe8 ffffffff ffffffff 00000000 c1000784 [ 667.031431] 1fc0: 00000000 c1079e30 00000000 c11f4494 c1103c58 c1079e2c c1108800 8000406a [ 667.039556] 1fe0: 410fc073 00000000 00000000 c1101ff8 8000807c c1000a7c 00000000 00000000 [ 667.047698] [<c028c0f4>] (kmem_cache_alloc) from [<c0895b9c>] (__build_skb+0x38/0xa0) [ 667.055485] [<c0895b9c>] (__build_skb) from [<c0895c24>] (build_skb+0x20/0x78) [ 667.062671] [<c0895c24>] (build_skb) from [<c071b4a0>] (mtk_napi_rx+0x184/0x568) [ 667.070024] [<c071b4a0>] (mtk_napi_rx) from [<c08ae4d8>] (net_rx_action+0x148/0x400) [ 667.077722] [<c08ae4d8>] (net_rx_action) from [<c0101614>] (__do_softirq+0x12c/0x378) [ 667.085507] [<c0101614>] (__do_softirq) from [<c012c998>] (irq_exit+0xe4/0x15c) [ 667.092775] [<c012c998>] (irq_exit) from [<c01818f4>] (__handle_domain_irq+0x70/0xc4) [ 667.100561] [<c01818f4>] (__handle_domain_irq) from [<c01014a0>] (gic_handle_irq+0x5c/0xa0) [ 667.108861] [<c01014a0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 667.116293] Exception stack(0xc1101ef0 to 0xc1101f38) [ 667.121309] 1ee0: 00000000 000aeb14 1e4e9000 c011f8e0 [ 667.129436] 1f00: ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 e07fcc00 c1101f4c [ 667.137560] 1f20: c1101f50 c1101f40 c01098ac c01098b0 600f0013 ffffffff [ 667.144138] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 667.151494] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 667.159537] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 667.167150] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 667.174675] [<c016e6b0>] (cpu_startup_entry) from [<c0abcc9c>] (rest_init+0xc0/0xc4) [ 667.182376] [<c0abcc9c>] (rest_init) from [<c1000e28>] (start_kernel+0x3b8/0x3c4) [ 667.189815] Code: e121f00c e35a0000 0a000002 e5943014 (e79a3003) [ 667.195994] ---[ end trace 2a6dec570c221166 ]--- [ 667.200595] Kernel panic - not syncing: Fatal exception in interrupt [ 667.206914] CPU3: stopping [ 667.209614] CPU: 3 PID: 165 Comm: systemd-journal Tainted: G D 4.14.95-bpi-r2-main #49 [ 667.218596] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 667.224148] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 667.231845] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 667.239025] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 667.246376] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 667.253898] [<c01014e0>] (gic_handle_irq) from [<c010e614>] (__irq_usr+0x54/0x80) [ 667.261329] Exception stack(0xdde3ffb0 to 0xdde3fff8) [ 667.266346] ffa0: bec8224f 0000000a 00000001 000005e8 [ 667.274471] ffc0: 00000001 b6ccd97d bec81e28 bec81db0 00000000 bec82280 00000009 00000000 [ 667.282594] ffe0: b6d69010 bec81d90 b6cb24cb b6cb1a5e 200d0030 ffffffff [ 667.289161] CPU2: stopping [ 667.291854] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 4.14.95-bpi-r2-main #49 [ 667.300146] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 667.305685] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 667.313380] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 667.320559] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 667.327909] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 667.335431] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 667.342861] Exception stack(0xde945f38 to 0xde945f80) [ 667.347875] 5f20: 00000000 000611a4 [ 667.356000] 5f40: 1e50b000 c011f8e0 ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 [ 667.364125] 5f60: 00000000 de945f94 de945f98 de945f88 c01098ac c01098b0 600f0013 ffffffff [ 667.372253] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 667.379605] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 667.387646] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 667.395254] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 667.402778] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194) [ 667.411766] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec) [ 667.419113] CPU1: stopping [ 667.421805] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 4.14.95-bpi-r2-main #49 [ 667.430098] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 667.435639] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 667.443334] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 667.450513] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 667.457863] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 667.465385] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 667.472816] Exception stack(0xde943f38 to 0xde943f80) [ 667.477831] 3f20: 00000000 00071d18 [ 667.485956] 3f40: 1e4fa000 c011f8e0 ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 [ 667.494081] 3f60: 00000000 de943f94 de943f98 de943f88 c01098ac c01098b0 680e0013 ffffffff [ 667.502208] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 667.509558] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 667.517598] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 667.525207] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 667.532730] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194) [ 667.541717] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec) [ 667.549071] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ============================================================================================================= [ 1851.777568] Internal err or: Oops: 5 [#1] SMP ARM [ 1851.782161] Modules linked in: xt_addrtype veth pppoe pppox ppp_generic slhc sit tunnel4 ip_tunnel brid ge ip6t_REJECT nf_reject_ipv6 xt_mac ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables i pt_REJECT nf_reject_ipv4 xt_limit xt_iprange xt_recent xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_m asquerade_ipv4 xt_nat tun xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con ntrack xt_mark iptable_mangle mtk_thermal thermal_sys mt6577_auxadc pwm_mediatek spi_mt65xx evbug mtk_pmic _keys rtc_mt6397 nvmem_mtk_efuse nfsd ip_tables x_tables ipv6 [last unloaded: mt76] [ 1851.835334] CPU: 0 PID: 3428 Comm: SoftwareVsyncTh Not tainted 4.14.95-bpi-r2-main #49 [ 1851.843192] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1851.848722] task: da788600 task.stack: da77e000 [ 1851.853226] PC is at kmem_cache_alloc+0xd0/0x21c [ 1851.857807] LR is at 0x6fc3d [ 1851.860663] pc : [<c028c0f4>] lr : [<0006fc3d>] psr: 20070113 [ 1851.866881] sp : da77fd70 ip : a0070113 fp : da77fda4 [ 1851.872064] r10: 00000001 r9 : de950900 r8 : c08958f0 [ 1851.877249] r7 : de801cc0 r6 : 01080020 r5 : c1226790 r4 : de801cc0 [ 1851.883727] r3 : 00000000 r2 : df5a39d4 r1 : 1e4e9000 r0 : 0006fc3c [ 1851.890207] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 1851.897290] Control: 10c5387d Table: 9bba006a DAC: 00000051 [ 1851.902992] Process SoftwareVsyncTh (pid: 3428, stack limit = 0xda77e218) [ 1851.909729] Stack: (0xda77fd70 to 0xda780000) [ 1851.914054] fd60: 1e4e9000 c019d1cc 20070193 01080020 [ 1851.922175] fd80: 00000080 00000000 de801cc0 01080020 00000000 00000000 da77fdd4 da77fda8 [ 1851.930302] fda0: c08958f0 c028c030 00000000 df59eb9c 01080020 db7b8300 c119f040 db7b8314 [ 1851.938428] fdc0: 0000001c 00000000 da77fe0c da77fdd8 c08b965c c08958a8 da77fdfc da77fde8 [ 1851.946553] fde0: c0149494 db7b8300 db7b8318 00000001 db7b8314 00000003 00000000 ffffe000 [ 1851.954678] fe00: da77fe34 da77fe10 c08be820 c08b961c c11bd17c 00000100 c08be68c db7b8330 [ 1851.962803] fe20: 00000200 00000000 da77fe6c da77fe38 c019bdc4 c08be698 c019db48 df5a4140 [ 1851.970928] fe40: 00000000 2617d280 000001af db7b8330 df59c440 da77fea8 c11bd17c 00000200 [ 1851.979053] fe60: da77fea4 da77fe70 c019c008 c019bd84 df5a4140 c1103c6c 00000000 da77fea4 [ 1851.987179] fe80: df59c440 da77fea4 c1102d00 c10b3440 1e4e9000 00000282 da77feec da77fea8 [ 1851.995304] fea0: c019c158 c019bf20 00000000 df59d5f8 da77ff0c da77fec0 c019ea04 c01af384 [ 1852.003429] fec0: a54fec74 000001af c1102084 00000001 c11bcd08 00000100 00000002 ffffe000 [ 1852.011554] fee0: da77ff4c da77fef0 c0101614 c019c090 00000010 de819000 e1003000 00400040 [ 1852.019679] ff00: c1102d00 00025e29 c0b023a8 0000000a c11f4c40 c1103c6c c10af310 c1102080 [ 1852.027804] ff20: c0186bf8 c10b7b3c 00000000 00000000 00000001 de819000 e1003000 6e5f9d25 [ 1852.035929] ff40: da77ff5c da77ff50 c012c998 c01014f4 da77ff84 da77ff60 c01818f4 c012c8c0 [ 1852.044054] ff60: c11043f8 c1168010 e100200c e1002000 da77ffb0 e1003000 da77ffac da77ff88 [ 1852.052178] ff80: c01014a0 c0181890 b6d8875e 20070030 ffffffff 10c5387d 10c5387d a54fec30 [ 1852.060303] ffa0: 00000000 da77ffb0 c010e614 c0101450 00000b98 000003e8 b6d88688 0002eaba [ 1852.068427] ffc0: a556b64c a54fec30 b6f86ce8 a54fec70 a54fec44 a54fec30 6e5f9d25 00000000 [ 1852.076551] ffe0: 2d4ca952 a54febf0 b361d339 b6d8875e 20070030 ffffffff 00000000 00000000 [ 1852.084692] [<c028c0f4>] (kmem_cache_alloc) from [<c08958f0>] (__alloc_skb+0x54/0x140) [ 1852.092568] [<c08958f0>] (__alloc_skb) from [<c08b965c>] (__neigh_notify+0x4c/0xdc) [ 1852.100181] [<c08b965c>] (__neigh_notify) from [<c08be820>] (neigh_timer_handler+0x194/0x278) [ 1852.108657] [<c08be820>] (neigh_timer_handler) from [<c019bdc4>] (call_timer_fn+0x4c/0x19c) [ 1852.116959] [<c019bdc4>] (call_timer_fn) from [<c019c008>] (expire_timers+0xf4/0x170) [ 1852.124742] [<c019c008>] (expire_timers) from [<c019c158>] (run_timer_softirq+0xd4/0x1cc) [ 1852.132870] [<c019c158>] (run_timer_softirq) from [<c0101614>] (__do_softirq+0x12c/0x378) [ 1852.140999] [<c0101614>] (__do_softirq) from [<c012c998>] (irq_exit+0xe4/0x15c) [ 1852.148266] [<c012c998>] (irq_exit) from [<c01818f4>] (__handle_domain_irq+0x70/0xc4) [ 1852.156049] [<c01818f4>] (__handle_domain_irq) from [<c01014a0>] (gic_handle_irq+0x5c/0xa0) [ 1852.164349] [<c01014a0>] (gic_handle_irq) from [<c010e614>] (__irq_usr+0x54/0x80) [ 1852.171779] Exception stack(0xda77ffb0 to 0xda77fff8) [ 1852.176795] ffa0: 00000b98 000003e8 b6d88688 0002eaba [ 1852.184920] ffc0: a556b64c a54fec30 b6f86ce8 a54fec70 a54fec44 a54fec30 6e5f9d25 00000000 [ 1852.193043] ffe0: 2d4ca952 a54febf0 b361d339 b6d8875e 20070030 ffffffff [ 1852.199615] Code: e121f00c e35a0000 0a000002 e5943014 (e79a3003) [ 1852.205765] ---[ end trace eeb67939bd037a21 ]--- [ 1852.210418] Kernel panic - not syncing: Fatal exception in interrupt [ 1852.216750] CPU2: stopping [ 1852.219449] CPU: 2 PID: 164 Comm: systemd-journal Tainted: G D 4.14.95-bpi-r2-main #49 [ 1852.228431] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1852.233982] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1852.241680] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1852.248860] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1852.256212] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1852.263734] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1852.271165] Exception stack(0xdde57eb0 to 0xdde57ef8) [ 1852.276183] 7ea0: 004e6190 beffffff 00000055 00000010 [ 1852.284308] 7ec0: ffffe000 00000051 004e6180 80808080 deb6d01c fefefeff 00000ff0 dde57f34 [ 1852.292431] 7ee0: 00000fe0 dde57f00 c04a9fd4 c0aa688c 20000013 ffffffff [ 1852.299004] [<c010e24c>] (__irq_svc) from [<c0aa688c>] (__get_user_4+0x0/0x20) [ 1852.306177] CPU1: stopping [ 1852.308869] CPU: 1 PID: 402 Comm: rs:main Q:Reg Tainted: G D 4.14.95-bpi-r2-main #49 [ 1852.317679] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1852.323219] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1852.330914] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1852.338092] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1852.345443] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1852.352964] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1852.360395] Exception stack(0xdc0d5da0 to 0xdc0d5de8) [ 1852.365413] 5da0: c1232c48 a0030013 00000007 00004fbe c1232c48 dec3a000 00000000 00000076 [ 1852.373537] 5dc0: 00000fff dcd36400 dc18a076 dc0d5dfc dc0d5e00 dc0d5df0 c052f398 c0ac2f24 [ 1852.381658] 5de0: 80030013 ffffffff [ 1852.385127] [<c010e24c>] (__irq_svc) from [<c0ac2f24>] (_raw_spin_unlock_irqrestore+0x30/0x34) [ 1852.393686] [<c0ac2f24>] (_raw_spin_unlock_irqrestore) from [<c052f398>] (uart_write+0x128/0x194) [ 1852.402505] [<c052f398>] (uart_write) from [<c0514660>] (n_tty_write+0x1c8/0x454) [ 1852.409942] [<c0514660>] (n_tty_write) from [<c05103c4>] (tty_write+0x19c/0x378) [ 1852.417296] [<c05103c4>] (tty_write) from [<c029fd28>] (__vfs_write+0x38/0x130) [ 1852.424563] [<c029fd28>] (__vfs_write) from [<c029ffb4>] (vfs_write+0xb4/0x178) [ 1852.431829] [<c029ffb4>] (vfs_write) from [<c02a01a8>] (SyS_write+0x4c/0xa0) [ 1852.438833] [<c02a01a8>] (SyS_write) from [<c0108c80>] (ret_fast_syscall+0x0/0x54) [ 1852.446352] CPU3: stopping [ 1852.449046] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 4.14.95-bpi-r2-main #49 [ 1852.457338] Hardware name: Mediatek Cortex-A7 (Device Tree) [ 1852.462878] [<c0113358>] (unwind_backtrace) from [<c010d580>] (show_stack+0x20/0x24) [ 1852.470574] [<c010d580>] (show_stack) from [<c0aa8e18>] (dump_stack+0x94/0xa8) [ 1852.477753] [<c0aa8e18>] (dump_stack) from [<c011124c>] (handle_IPI+0x358/0x378) [ 1852.485103] [<c011124c>] (handle_IPI) from [<c01014e0>] (gic_handle_irq+0x9c/0xa0) [ 1852.492625] [<c01014e0>] (gic_handle_irq) from [<c010e24c>] (__irq_svc+0x6c/0x90) [ 1852.500055] Exception stack(0xde947f38 to 0xde947f80) [ 1852.505071] 7f20: 00000000 000fa8e8 [ 1852.513195] 7f40: 1e51c000 c011f8e0 ffffe000 c1103ccc c1103c6c c11bc85f c0d2c0e0 00000001 [ 1852.521320] 7f60: 00000000 de947f94 de947f98 de947f88 c01098ac c01098b0 60000013 ffffffff [ 1852.529447] [<c010e24c>] (__irq_svc) from [<c01098b0>] (arch_cpu_idle+0x48/0x4c) [ 1852.536799] [<c01098b0>] (arch_cpu_idle) from [<c0ac2d88>] (default_idle_call+0x30/0x3c) [ 1852.544841] [<c0ac2d88>] (default_idle_call) from [<c016e3b0>] (do_idle+0xec/0x150) [ 1852.552451] [<c016e3b0>] (do_idle) from [<c016e6b0>] (cpu_startup_entry+0x28/0x2c) [ 1852.559974] [<c016e6b0>] (cpu_startup_entry) from [<c0110c60>] (secondary_start_kernel+0x170/0x194) [ 1852.568962] [<c0110c60>] (secondary_start_kernel) from [<801018ec>] (0x801018ec) [ 1852.576318] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb 2019-02-05 15:02 ` [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb Frank Wunderlich @ 2019-02-05 18:27 ` Greg KH 2019-02-06 13:53 ` Aw: " Frank Wunderlich 0 siblings, 1 reply; 4+ messages in thread From: Greg KH @ 2019-02-05 18:27 UTC (permalink / raw) To: Frank Wunderlich; +Cc: stable On Tue, Feb 05, 2019 at 04:02:50PM +0100, Frank Wunderlich wrote: > Hi, > > i've noticed panics on my bananapi-r2 with kernel 4.14.95, but can't > locate them in detail...all is pointing to kmem_cache_alloc and > build_skb, but over different ways. You should try to see if this happens for 4.19, or best yet, 4.20 or 5.0-rc. If so, then ask about this on the networking mailing list, nothing we can do here about that. And, if it does work properly, and you do find an easy reproducer, try using 'git bisect' to find the fix and let us know would be great. thanks, greg k-h ^ permalink raw reply [flat|nested] 4+ messages in thread
* Aw: Re: [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb 2019-02-05 18:27 ` Greg KH @ 2019-02-06 13:53 ` Frank Wunderlich 2019-02-06 16:12 ` Greg KH 0 siblings, 1 reply; 4+ messages in thread From: Frank Wunderlich @ 2019-02-06 13:53 UTC (permalink / raw) To: Greg KH; +Cc: stable > Gesendet: Dienstag, 05. Februar 2019 um 19:27 Uhr > Von: "Greg KH" <gregkh@linuxfoundation.org> > If so, then ask about this on the networking mailing list, > nothing we can do here about that. Hi Greg, how do you think problem is in network (because of build_skb)? imho kmem_cache_alloc is mm-related i've posted to stable because this issue seems to be caused by stable-updates between 4.14.78 and 4.14.95 and maybe anybody here have seen this before i try to reproduce in kernel-versions between, on 4.14.95 it happens after ~ 30 minutes of having device running regards Frank ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Re: [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb 2019-02-06 13:53 ` Aw: " Frank Wunderlich @ 2019-02-06 16:12 ` Greg KH 0 siblings, 0 replies; 4+ messages in thread From: Greg KH @ 2019-02-06 16:12 UTC (permalink / raw) To: Frank Wunderlich; +Cc: stable On Wed, Feb 06, 2019 at 02:53:17PM +0100, Frank Wunderlich wrote: > > Gesendet: Dienstag, 05. Februar 2019 um 19:27 Uhr > > Von: "Greg KH" <gregkh@linuxfoundation.org> > > If so, then ask about this on the networking mailing list, > > nothing we can do here about that. > > Hi Greg, > > how do you think problem is in network (because of build_skb)? > imho kmem_cache_alloc is mm-related > > i've posted to stable because this issue seems to be caused by > stable-updates between 4.14.78 and 4.14.95 and maybe anybody here have > seen this before I would cc: the subsystems you think are responsible the stable@ list is usually for patches to be applied to the next stable release, not for debugging old problems. And again, please try 4.20 and see if that fixes it. thanks, greg k-h ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-02-06 16:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <trinity-cc5e013b-ac22-4e47-a902-57a6cac67965-1549350557264@3c-app-gmx-bs52>
2019-02-05 15:02 ` [BUG] 4.14.95 panic in kmem_cache_alloc / build_skb Frank Wunderlich
2019-02-05 18:27 ` Greg KH
2019-02-06 13:53 ` Aw: " Frank Wunderlich
2019-02-06 16:12 ` Greg KH
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.