From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753156Ab3LBRti (ORCPT ); Mon, 2 Dec 2013 12:49:38 -0500 Received: from mx1.redhat.com ([209.132.183.28]:53077 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752992Ab3LBRtf (ORCPT ); Mon, 2 Dec 2013 12:49:35 -0500 Date: Mon, 2 Dec 2013 12:49:13 -0500 From: Dave Jones To: Gu Zheng Cc: Kristian Nielsen , Benjamin LaHaise , Kent Overstreet , Linux Kernel , Sasha Levin Subject: Re: GPF in aio_migratepage Message-ID: <20131202174913.GA18853@redhat.com> Mail-Followup-To: Dave Jones , Gu Zheng , Kristian Nielsen , Benjamin LaHaise , Kent Overstreet , Linux Kernel , Sasha Levin References: <20131126032645.GA32301@redhat.com> <20131126060132.GA6400@redhat.com> <20131126071953.GE9244@kmo-pixel> <20131126152337.GL15489@kvack.org> <87d2lh6h92.fsf@frigg.knielsen-hq.org> <529C5CA6.6090708@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <529C5CA6.6090708@cn.fujitsu.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 02, 2013 at 06:10:46PM +0800, Gu Zheng wrote: > Hi Kristian, Dave, > > Could you please help to check whether the following patch can fix this issue? This introduces some locking bugs.. [ 222.327950] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:616 [ 222.328004] in_atomic(): 1, irqs_disabled(): 0, pid: 12794, name: trinity-child1 [ 222.328044] 1 lock held by trinity-child1/12794: [ 222.328072] #0: (&(&mapping->private_lock)->rlock){+.+...}, at: [] aio_free_ring+0x44/0x160 [ 222.328147] CPU: 1 PID: 12794 Comm: trinity-child1 Not tainted 3.13.0-rc2+ #12 [ 222.328268] 0000000000000268 ffff880229517d68 ffffffff8173bc52 0000000000000000 [ 222.328320] ffff880229517d90 ffffffff8108ad95 ffff880223b6acd0 0000000000000000 [ 222.328370] 0000000000000000 ffff880229517e08 ffffffff81741cf3 ffff880229517dc0 [ 222.328421] Call Trace: [ 222.328443] [] dump_stack+0x4e/0x7a [ 222.328475] [] __might_sleep+0x175/0x200 [ 222.328510] [] mutex_lock_nested+0x33/0x400 [ 222.328545] [] unmap_mapping_range+0x68/0x170 [ 222.328582] [] truncate_pagecache+0x35/0x60 [ 222.328617] [] truncate_setsize+0x12/0x20 [ 222.328651] [] aio_free_ring+0x99/0x160 [ 222.328684] [] SyS_io_setup+0xef1/0xf00 [ 222.328717] [] tracesys+0xdd/0xe2 [ 222.328769] ====================================================== [ 222.328804] [ INFO: possible circular locking dependency detected ] [ 222.328838] 3.13.0-rc2+ #12 Not tainted [ 222.328862] ------------------------------------------------------- [ 222.328896] trinity-child1/12794 is trying to acquire lock: [ 222.328928] (&mapping->i_mmap_mutex){+.+...}, at: [] unmap_mapping_range+0x68/0x170 [ 222.328987] but task is already holding lock: [ 222.329020] (&(&mapping->private_lock)->rlock){+.+...}, at: [] aio_free_ring+0x44/0x160 [ 222.329081] which lock already depends on the new lock. [ 222.329125] the existing dependency chain (in reverse order) is: [ 222.329166] -> #2 (&(&mapping->private_lock)->rlock){+.+...}: [ 222.329211] [] lock_acquire+0x93/0x1c0 [ 222.329248] [] _raw_spin_lock+0x40/0x80 [ 222.329285] [] __set_page_dirty_buffers+0x2d/0xb0 [ 222.331243] [] set_page_dirty+0x3a/0x60 [ 222.334437] [] unmap_single_vma+0x62f/0x830 [ 222.337633] [] unmap_vmas+0x49/0x90 [ 222.340819] [] unmap_region+0x9d/0x110 [ 222.343968] [] do_munmap+0x226/0x3b0 [ 222.346689] [] vm_munmap+0x44/0x60 [ 222.349741] [] SyS_munmap+0x22/0x30 [ 222.352758] [] tracesys+0xdd/0xe2 [ 222.355735] -> #1 (&(ptlock_ptr(page))->rlock#2){+.+...}: [ 222.361611] [] lock_acquire+0x93/0x1c0 [ 222.364589] [] _raw_spin_lock+0x40/0x80 [ 222.367200] [] __page_check_address+0x98/0x160 [ 222.370168] [] page_mkclean+0xfe/0x1c0 [ 222.373120] [] clear_page_dirty_for_io+0x60/0x100 [ 222.376076] [] mpage_submit_page+0x47/0x80 [ 222.379015] [] mpage_process_page_bufs+0x110/0x130 [ 222.381955] [] mpage_prepare_extent_to_map+0x22b/0x2f0 [ 222.384895] [] ext4_writepages+0x4ef/0x1050 [ 222.387839] [] do_writepages+0x21/0x50 [ 222.390786] [] __filemap_fdatawrite_range+0x59/0x60 [ 222.393747] [] filemap_write_and_wait_range+0x2d/0x70 [ 222.396729] [] ext4_sync_file+0xba/0x4d0 [ 222.399714] [] do_fsync+0x51/0x80 [ 222.402317] [] SyS_fsync+0x10/0x20 [ 222.405240] [] tracesys+0xdd/0xe2 [ 222.407760] -> #0 (&mapping->i_mmap_mutex){+.+...}: [ 222.413349] [] __lock_acquire+0x1786/0x1af0 [ 222.416127] [] lock_acquire+0x93/0x1c0 [ 222.418826] [] mutex_lock_nested+0x77/0x400 [ 222.421456] [] unmap_mapping_range+0x68/0x170 [ 222.424085] [] truncate_pagecache+0x35/0x60 [ 222.426696] [] truncate_setsize+0x12/0x20 [ 222.428955] [] aio_free_ring+0x99/0x160 [ 222.431509] [] SyS_io_setup+0xef1/0xf00 [ 222.434069] [] tracesys+0xdd/0xe2 [ 222.436308] other info that might help us debug this: [ 222.443857] Chain exists of: &mapping->i_mmap_mutex --> &(ptlock_ptr(page))->rlock#2 --> &(&mapping->private_lock)->rlock [ 222.451618] Possible unsafe locking scenario: [ 222.456831] CPU0 CPU1 [ 222.459413] ---- ---- [ 222.461958] lock(&(&mapping->private_lock)->rlock); [ 222.464505] lock(&(ptlock_ptr(page))->rlock#2); [ 222.467094] lock(&(&mapping->private_lock)->rlock); [ 222.469625] lock(&mapping->i_mmap_mutex); [ 222.472111] *** DEADLOCK *** [ 222.478392] 1 lock held by trinity-child1/12794: [ 222.480744] #0: (&(&mapping->private_lock)->rlock){+.+...}, at: [] aio_free_ring+0x44/0x160 [ 222.483240] stack backtrace: [ 222.488119] CPU: 1 PID: 12794 Comm: trinity-child1 Not tainted 3.13.0-rc2+ #12 [ 222.493016] ffffffff824cb110 ffff880229517c30 ffffffff8173bc52 ffffffff824a3f40 [ 222.495690] ffff880229517c70 ffffffff81737fed ffff880229517cc0 ffff8800a1e49d10 [ 222.498379] ffff8800a1e495d0 0000000000000001 0000000000000001 ffff8800a1e49d10 [ 222.501073] Call Trace: [ 222.503394] [] dump_stack+0x4e/0x7a [ 222.506080] [] print_circular_bug+0x200/0x20f [ 222.508781] [] __lock_acquire+0x1786/0x1af0 [ 222.511485] [] lock_acquire+0x93/0x1c0 [ 222.514197] [] ? unmap_mapping_range+0x68/0x170 [ 222.516594] [] ? unmap_mapping_range+0x68/0x170 [ 222.519307] [] mutex_lock_nested+0x77/0x400 [ 222.522028] [] ? unmap_mapping_range+0x68/0x170 [ 222.524752] [] ? unmap_mapping_range+0x68/0x170 [ 222.527445] [] unmap_mapping_range+0x68/0x170 [ 222.530113] [] truncate_pagecache+0x35/0x60 [ 222.532785] [] truncate_setsize+0x12/0x20 [ 222.535439] [] aio_free_ring+0x99/0x160 [ 222.538089] [] SyS_io_setup+0xef1/0xf00 [ 222.540725] [] tracesys+0xdd/0xe2