From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3430AC004D3 for ; Wed, 24 Oct 2018 12:48:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D3A742064A for ; Wed, 24 Oct 2018 12:48:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="My1u3/j1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3A742064A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-btrfs-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727204AbeJXVQy (ORCPT ); Wed, 24 Oct 2018 17:16:54 -0400 Received: from mail.kernel.org ([198.145.29.99]:55762 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727009AbeJXVQy (ORCPT ); Wed, 24 Oct 2018 17:16:54 -0400 Received: from mail-ua1-f41.google.com (mail-ua1-f41.google.com [209.85.222.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 41CBA2082E for ; Wed, 24 Oct 2018 12:48:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1540385333; bh=x5IxX4sgYmuHaQyvfQxOwxzr7YuzjsIP0JiYSr1r+4Y=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=My1u3/j1SK/PL5gKP8a0nPtjLBtGqs+PZ1V8QQyBUXOFvwk6BpE8KVI/hoGJHPUjS x8r48xsLD0QcI96E2f0wIHPL4EyPilFNwe9Ei3lF2C5fnz9jO2IZ5rMUAZPKungDx4 RV1Zs3yzkblE1dwnqeY3Otn7iW2InqQNqAxeKKeo= Received: by mail-ua1-f41.google.com with SMTP id o17so1795348uad.8 for ; Wed, 24 Oct 2018 05:48:53 -0700 (PDT) X-Gm-Message-State: AGRZ1gJiCyCbyA9oxfV31KqAEv4FsJ2DZ3SEetqNrA3QtEucYVafAT5X cM8Qqfxg32s07p6O0gzLHscAjAR8UXZxOsO0zRk= X-Google-Smtp-Source: AJdET5dDyfi0RQ67nYJ86e/Qx9Mv/dITJWY6hkdZWDGEPO0Ipk17my3fNBdZoa0HeoWWahHGVOFsR3BhXVPHuOKCroM= X-Received: by 2002:a9f:2c0b:: with SMTP id r11mr1110227uaj.100.1540385332317; Wed, 24 Oct 2018 05:48:52 -0700 (PDT) MIME-Version: 1.0 References: <20181022090946.1150-1-fdmanana@kernel.org> <20181024091303.20324-1-fdmanana@kernel.org> <20181024113717.fcz4klha7isb7r56@MacBook-Pro-91.local> <20181024123959.icp7ssu5sjhqg6c3@MacBook-Pro-91.local> In-Reply-To: <20181024123959.icp7ssu5sjhqg6c3@MacBook-Pro-91.local> From: Filipe Manana Date: Wed, 24 Oct 2018 13:48:40 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent To: Josef Bacik Cc: linux-btrfs , Filipe David Borba Manana Content-Type: text/plain; charset="UTF-8" Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On Wed, Oct 24, 2018 at 1:40 PM Josef Bacik wrote: > > On Wed, Oct 24, 2018 at 12:53:59PM +0100, Filipe Manana wrote: > > On Wed, Oct 24, 2018 at 12:37 PM Josef Bacik wrote: > > > > > > On Wed, Oct 24, 2018 at 10:13:03AM +0100, fdmanana@kernel.org wrote: > > > > From: Filipe Manana > > > > > > > > When we are writing out a free space cache, during the transaction commit > > > > phase, we can end up in a deadlock which results in a stack trace like the > > > > following: > > > > > > > > schedule+0x28/0x80 > > > > btrfs_tree_read_lock+0x8e/0x120 [btrfs] > > > > ? finish_wait+0x80/0x80 > > > > btrfs_read_lock_root_node+0x2f/0x40 [btrfs] > > > > btrfs_search_slot+0xf6/0x9f0 [btrfs] > > > > ? evict_refill_and_join+0xd0/0xd0 [btrfs] > > > > ? inode_insert5+0x119/0x190 > > > > btrfs_lookup_inode+0x3a/0xc0 [btrfs] > > > > ? kmem_cache_alloc+0x166/0x1d0 > > > > btrfs_iget+0x113/0x690 [btrfs] > > > > __lookup_free_space_inode+0xd8/0x150 [btrfs] > > > > lookup_free_space_inode+0x5b/0xb0 [btrfs] > > > > load_free_space_cache+0x7c/0x170 [btrfs] > > > > ? cache_block_group+0x72/0x3b0 [btrfs] > > > > cache_block_group+0x1b3/0x3b0 [btrfs] > > > > ? finish_wait+0x80/0x80 > > > > find_free_extent+0x799/0x1010 [btrfs] > > > > btrfs_reserve_extent+0x9b/0x180 [btrfs] > > > > btrfs_alloc_tree_block+0x1b3/0x4f0 [btrfs] > > > > __btrfs_cow_block+0x11d/0x500 [btrfs] > > > > btrfs_cow_block+0xdc/0x180 [btrfs] > > > > btrfs_search_slot+0x3bd/0x9f0 [btrfs] > > > > btrfs_lookup_inode+0x3a/0xc0 [btrfs] > > > > ? kmem_cache_alloc+0x166/0x1d0 > > > > btrfs_update_inode_item+0x46/0x100 [btrfs] > > > > cache_save_setup+0xe4/0x3a0 [btrfs] > > > > btrfs_start_dirty_block_groups+0x1be/0x480 [btrfs] > > > > btrfs_commit_transaction+0xcb/0x8b0 [btrfs] > > > > > > > > At cache_save_setup() we need to update the inode item of a block group's > > > > cache which is located in the tree root (fs_info->tree_root), which means > > > > that it may result in COWing a leaf from that tree. If that happens we > > > > need to find a free metadata extent and while looking for one, if we find > > > > a block group which was not cached yet we attempt to load its cache by > > > > calling cache_block_group(). However this function will try to load the > > > > inode of the free space cache, which requires finding the matching inode > > > > item in the tree root - if that inode item is located in the same leaf as > > > > the inode item of the space cache we are updating at cache_save_setup(), > > > > we end up in a deadlock, since we try to obtain a read lock on the same > > > > extent buffer that we previously write locked. > > > > > > > > So fix this by using the tree root's commit root when searching for a > > > > block group's free space cache inode item when we are attempting to load > > > > a free space cache. This is safe since block groups once loaded stay in > > > > memory forever, as well as their caches, so after they are first loaded > > > > we will never need to read their inode items again. For new block groups, > > > > once they are created they get their ->cached field set to > > > > BTRFS_CACHE_FINISHED meaning we will not need to read their inode item. > > > > > > > > Reported-by: Andrew Nelson > > > > Link: https://lore.kernel.org/linux-btrfs/CAPTELenq9x5KOWuQ+fa7h1r3nsJG8vyiTH8+ifjURc_duHh2Wg@mail.gmail.com/ > > > > Fixes: 9d66e233c704 ("Btrfs: load free space cache if it exists") > > > > Tested-by: Andrew Nelson > > > > Signed-off-by: Filipe Manana > > > > --- > > > > > > > > > > Now my goal is to see how many times I can get you to redo this thing. > > > > > > Why not instead just do > > > > > > if (btrfs_is_free_space_inode(inode)) > > > path->search_commit_root = 1; > > > > > > in read_locked_inode? That would be cleaner. If we don't want to do that for > > > the inode cache (I'm not sure if it's ok or not) we could just do > > > > > > if (root == fs_info->tree_root) > > > > We can't (not just that at least). > > Tried something like that, but we get into a BUG_ON when writing out > > the space cache for new block groups (created in the current > > transaction). > > Because at cache_save_setup() we have this: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/extent-tree.c?h=v4.19#n3342 > > > > Lookup for the inode in normal root, doesn't exist, create it then > > repeat - if still not found, BUG_ON. > > Could also make create_free_space_inode() return an inode pointer and > > make it call btrfs_iget(). > > > > Ah ok makes sense. Well in that case lets just make btrfs_read_locked_inode() > take a path, and allocate it in btrfs_iget, that'll remove the ugly > > if (path != in_path) You mean the following on top of v4: https://friendpaste.com/6XrGXb5p0RSJGixUFYouHg Not much different, just saves one such if statement. I'm ok with that. > > stuff. Thanks, > > Josef