From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751980AbZJKH6v (ORCPT ); Sun, 11 Oct 2009 03:58:51 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751523AbZJKH6u (ORCPT ); Sun, 11 Oct 2009 03:58:50 -0400 Received: from smtp1.linux-foundation.org ([140.211.169.13]:34022 "EHLO smtp1.linux-foundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751475AbZJKH6u (ORCPT ); Sun, 11 Oct 2009 03:58:50 -0400 Date: Sun, 11 Oct 2009 00:57:46 -0700 From: Andrew Morton To: Thomas Neumann Cc: linux-kernel@vger.kernel.org, xfs-masters@oss.sgi.com, Christoph Hellwig Subject: Re: deadlock with fallocate Message-Id: <20091011005746.56cd3cd4.akpm@linux-foundation.org> In-Reply-To: References: X-Mailer: Sylpheed 2.4.8 (GTK+ 2.12.5; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 07 Oct 2009 05:00:36 +0200 Thomas Neumann wrote: > Under 2.6.32rc1 I observe a very annoying deadlock in fallocate calls with > XFS. When running a very I/O intensive workload (using posix_fallocate and a > lot of mmaps) the kernel deadlocks semi-reproducible (ca. 60% of the time). > I included an excerpt from dmesg below, but I am not sure how useful it is > for debugging. > Is there any way to isolate the problem? Unfortunately it is not 100% > reproducible and it takes a while to reach the deadlock. I'm wondering how serious this bug is. Will legacy applications fail on newer kernels? Or is it the case that only recently-written applications which utilise new kernel functionality will hit this bug? If the former then it's time to run around in panic looking for changes to revert. [trace retained for the newly cc'ed] > [36600.700131] INFO: task xfsconvertd/0:411 blocked for more than 120 > seconds. > [36600.700138] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [36600.700143] xfsconvertd/0 D 0000000000000000 0 411 2 > 0x00000000 > [36600.700153] ffff88007b21d3e0 0000000000000046 ffff88007d4e8c40 > ffff88007b21dfd8 > [36600.700163] ffff88007adfdb40 0000000000015980 0000000000015980 > ffff88007b21dfd8 > [36600.700171] 0000000000015980 ffff88007b21dfd8 0000000000015980 > ffff88007adfdf00 > [36600.700180] Call Trace: > [36600.700195] [] io_schedule+0x42/0x60 > [36600.700204] [] sync_page+0x35/0x50 > [36600.700211] [] __wait_on_bit+0x55/0x80 > [36600.700218] [] ? sync_page+0x0/0x50 > [36600.700225] [] wait_on_page_bit+0x70/0x80 > [36600.700233] [] ? wake_bit_function+0x0/0x30 > [36600.700242] [] shrink_page_list+0x3d8/0x550 > [36600.700249] [] ? finish_wait+0x5e/0x80 > [36600.700258] [] ? congestion_wait+0x7a/0x90 > [36600.700265] [] ? autoremove_wake_function+0x0/0x40 > [36600.700273] [] shrink_inactive_list+0x6b6/0x700 > [36600.700281] [] ? wake_affine+0x220/0x240 > [36600.700289] [] ? __enqueue_entity+0x80/0x90 > [36600.700296] [] shrink_list+0x51/0xb0 > [36600.700303] [] shrink_zone+0x1ea/0x200 > [36600.700311] [] shrink_zones+0x63/0xf0 > [36600.700318] [] do_try_to_free_pages+0x70/0x280 > [36600.700325] [] try_to_free_pages+0x9c/0xc0 > [36600.700332] [] ? isolate_pages_global+0x0/0x50 > [36600.700340] [] __alloc_pages_slowpath+0x232/0x520 > [36600.700347] [] __alloc_pages_nodemask+0x146/0x180 > [36600.700355] [] alloc_pages_current+0x87/0xd0 > [36600.700363] [] allocate_slab+0x11c/0x1b0 > [36600.700369] [] new_slab+0x2b/0x190 > [36600.700376] [] __slab_alloc+0x121/0x230 > [36600.700422] [] ? kmem_zone_alloc+0x5d/0xd0 [xfs] > [36600.700452] [] ? kmem_zone_alloc+0x5d/0xd0 [xfs] > [36600.700459] [] kmem_cache_alloc+0xf0/0x130 > [36600.700489] [] kmem_zone_alloc+0x5d/0xd0 [xfs] > [36600.700519] [] kmem_zone_zalloc+0x19/0x50 [xfs] > [36600.700550] [] _xfs_trans_alloc+0x2f/0x70 [xfs] > [36600.700580] [] xfs_trans_alloc+0x92/0xa0 [xfs] > [36600.700588] [] ? __dec_zone_page_state+0x30/0x40 > [36600.700619] [] xfs_iomap_write_unwritten+0x71/0x200 > [xfs] > [36600.700627] [] ? mempool_free_slab+0x12/0x20 > [36600.700634] [] ? mempool_free+0x8a/0xa0 > [36600.700663] [] ? xfs_end_bio_unwritten+0x0/0x80 [xfs] > [36600.700692] [] xfs_end_bio_unwritten+0x65/0x80 [xfs] > [36600.700701] [] run_workqueue+0xb7/0x190 > [36600.700707] [] worker_thread+0x96/0xf0 > [36600.700714] [] ? autoremove_wake_function+0x0/0x40 > [36600.700721] [] ? worker_thread+0x0/0xf0 > [36600.700727] [] kthread+0x8e/0xa0 > [36600.700735] [] child_rip+0xa/0x20 > [36600.700742] [] ? kthread+0x0/0xa0 > [36600.700748] [] ? child_rip+0x0/0x20 > [36600.700784] INFO: task dataload:3947 blocked for more than 120 seconds. > [36600.700788] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [36600.700792] dataload D 0000000000000001 0 3947 2121 0x00000000 > [36600.700801] ffff880074f37ce8 0000000000000082 0000000000000086 > ffff880074f37fd8 > [36600.700810] ffff88007b0316d0 0000000000015980 0000000000015980 > ffff880074f37fd8 > [36600.700818] 0000000000015980 ffff880074f37fd8 0000000000015980 > ffff88007b031a90 > [36600.700826] Call Trace: > [36600.700859] [] xfs_ioend_wait+0x85/0xc0 [xfs] > [36600.700868] [] ? autoremove_wake_function+0x0/0x40 > [36600.700899] [] xfs_setattr+0x85d/0xb20 [xfs] > [36600.700930] [] ? xfs_trans_unlocked_item+0x38/0x50 > [xfs] > [36600.700960] [] xfs_vn_fallocate+0xed/0x100 [xfs] > [36600.700968] [] ? default_spin_lock_flags+0x9/0x10 > [36600.700976] [] ? _spin_lock_irqsave+0x2f/0x50 > [36600.700984] [] do_fallocate+0xfd/0x110 > [36600.700990] [] sys_fallocate+0x49/0x70 > [36600.700998] [] system_call_fastpath+0x16/0x1b > > I have also opened a bug here > > http://oss.sgi.com/bugzilla/show_bug.cgi?id=852 >