From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753509AbbIKR3S (ORCPT ); Fri, 11 Sep 2015 13:29:18 -0400 Received: from e37.co.us.ibm.com ([32.97.110.158]:37775 "EHLO e37.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752890AbbIKR3Q (ORCPT ); Fri, 11 Sep 2015 13:29:16 -0400 X-Helo: d03dlp03.boulder.ibm.com X-MailFrom: paulmck@linux.vnet.ibm.com X-RcptTo: linux-kernel@vger.kernel.org Date: Fri, 11 Sep 2015 10:18:49 -0700 From: "Paul E. McKenney" To: Linus Walleij Cc: Josh Triplett , "linux-kernel@vger.kernel.org" Subject: Re: RCU explosion on ARM Integrator Message-ID: <20150911171849.GX4029@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15091117-0025-0000-0000-00001C56D17E Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 11, 2015 at 01:24:56PM +0200, Linus Walleij wrote: > Hi RCU folks, > > this happened to me when running the iozone throughput benchmark > on the ARM Integrator, I wonder if I should take this platform for a ride on > the RCU torture test or similar? Looks a bit instable :/ You got a pagefault in rcu_check_callbacks(). Congratulations, that -is- an accomplishment! ;-) I haven't seen anything like this recently. Is this reproducible? If so, and if it was stable on some previous release, a bisection would be helpful. Otherwise, it looks like this blew up just after returning from a function call. If you could map back to the source code, let me know what version you are running, send me a disassembly of rcu_check_callbacks(), and supply a .config, I can take a look and see if I can provide any additional information. Or, for that matter, a fix. On the other hand, if this is a new port, things to be suspicious of include correct masking of interrupts, consistent reporting of the number of CPUs, and of course memory mapping. Thanx, Paul > Yours, > Linus Walleij > > root@integrator:/ iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test > Iozone: Performance Test of File I/O > Version $Revision: 3.430 $ > Compiled for 32 bit mode. > Build: linux-arm > > Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins > Al Slater, Scott Rhine, Mike Wisner, Ken Goss > Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, > Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, > Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, > Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, > Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, > Vangel Bojaxhi, Ben England, Vikentsi Lapa. > > Run began: Thu Jan 1 01:16:39 1970 > > Auto Mode > Cross over of record size disabled. > File size set to 20480 kB > O_DIRECT feature enabled > Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test > Output is in kBytes/sec > Time Resolution = 0.000016 seconds. > Processor cache size set to 1024 kBytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random > random bkwd record stride > kB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread > freread > 20480 4 54 56 57 57 57 > 24 > 20480 8 56 57 58 58 58 > 34 > 20480 16 56 58 59 59Unable to > handle kernel paging request at virtual address 807b7cac > pgd = c6404000 > [807b7cac] *pgd=00000000 > Internal error: Oops: 5 [#1] PREEMPT ARM > Modules linked in: > CPU: 0 PID: 110 Comm: iozone Not tainted 4.2.0-11142-gb0a1ea51bda4-dirty #3 > Hardware name: ARM Integrator/AP (Device Tree) > task: c6b45540 ti: c6420000 task.ti: c6420000 > PC is at rcu_check_callbacks+0x318/0x850 > LR is at rcu_check_callbacks+0x310/0x850 > pc : [] lr : [] psr: 60000093 > sp : c64218c0 ip : c07b8038 fp : c07b8920 > r10: 807b7ca8 r9 : 00000001 r8 : c07b79f8 > r7 : c07b2110 r6 : c07b2118 r5 : c07b7ca8 r4 : c07b8004 > r3 : c07b8038 r2 : c07b8038 r1 : c07b8004 r0 : 00000000 > Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none > Control: 0005317f Table: 06404000 DAC: 00000051 > Process iozone (pid: 110, stack limit = 0xc6420190) > Stack: (0xc64218c0 to 0xc6422000) > 18c0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 18e0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1900: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1920: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1940: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1960: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1980: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 19a0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 19c0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 19e0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1a00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1a20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1a40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1a60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1a80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1aa0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ac0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ae0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1b00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1b20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1b40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1b60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1b80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ba0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1bc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1be0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1c00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1c20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1c40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1c60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1c80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ca0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1cc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ce0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1d00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1d20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1d40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1d60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1d80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1da0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1dc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1de0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1e00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1e20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1e40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1e60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1e80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ea0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ec0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1ee0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1f00: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1f20: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1f40: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1f60: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1f80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1fa0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1fe0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > [] (rcu_check_callbacks) from [] > (update_process_times+0x38/0x60) > [] (update_process_times) from [] > (tick_sched_timer+0x4c/0x98) > [] (tick_sched_timer) from [] > (__hrtimer_run_queues.constprop.36+0x124/0x1d4) > [] (__hrtimer_run_queues.constprop.36) from [] > (hrtimer_interrupt+0x9c/0x270) > [] (hrtimer_interrupt) from [] > (integrator_timer_interrupt+0x20/0x2c) > [] (integrator_timer_interrupt) from [] > (handle_irq_event_percpu+0x78/0x144) > [] (handle_irq_event_percpu) from [] > (handle_irq_event+0x54/0x8c) > [] (handle_irq_event) from [] (handle_level_irq+0xdc/0x168) > [] (handle_level_irq) from [] (generic_handle_irq+0x2c/0x40) > [] (generic_handle_irq) from [] > (__handle_domain_irq+0x5c/0xd4) > [] (__handle_domain_irq) from [] (fpga_handle_irq+0x84/0xc4) > [] (fpga_handle_irq) from [] (__irq_svc+0x44/0x78) > Exception stack(0xc6421a68 to 0xc6421ab0) > 1a60: ???????? ???????? ???????? ???????? ???????? ???????? > 1a80: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? > 1aa0: ???????? ???????? ???????? ???????? > [] (__irq_svc) from [] > (__slab_alloc.isra.90.constprop.92+0x124/0x2ec) > [] (__slab_alloc.isra.90.constprop.92) from [] > (kmem_cache_alloc+0xf0/0x130) > [] (kmem_cache_alloc) from [] (mempool_alloc+0x44/0x1b8) > [] (mempool_alloc) from [] (bio_alloc_bioset+0x128/0x214) > [] (bio_alloc_bioset) from [] (bio_clone_bioset+0xf4/0x2cc) > [] (bio_clone_bioset) from [] (blk_queue_split+0x1a4/0x438) > [] (blk_queue_split) from [] (blk_queue_bio+0x28/0x284) > [] (blk_queue_bio) from [] (generic_make_request+0xb8/0xdc) > [] (generic_make_request) from [] (submit_bio+0x80/0x16c) > [] (submit_bio) from [] (__blockdev_direct_IO+0x11e4/0x1a4c) > [] (__blockdev_direct_IO) from [] (ext2_direct_IO+0x54/0x94) > [] (ext2_direct_IO) from [] > (generic_file_direct_write+0x94/0x1cc) > [] (generic_file_direct_write) from [] > (__generic_file_write_iter+0xb4/0x21c) > [] (__generic_file_write_iter) from [] > (generic_file_write_iter+0x108/0x2a0) > [] (generic_file_write_iter) from [] (__vfs_write+0xb0/0xe4) > [] (__vfs_write) from [] (vfs_write+0x90/0x164) > [] (vfs_write) from [] (SyS_write+0x44/0x9c) > [] (SyS_write) from [] (ret_fast_syscall+0x0/0x38) > Code: bad PC value > ---[ end trace 59d7580d1dfe574e ]--- > Kernel panic - not syncing: Fatal exception in interrupt > ---[ end Kernel panic - not syncing: Fatal exception in interrupt > > > Yours, > Linus Walleij >