From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mondschein.lichtvoll.de ([194.150.191.11]:43772 "EHLO mail.lichtvoll.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750817AbaHOKZk convert rfc822-to-8bit (ORCPT ); Fri, 15 Aug 2014 06:25:40 -0400 From: Martin Steigerwald To: Valdis Voronin Cc: linux-btrfs@vger.kernel.org Subject: Re: btrfs 3.14-2-amd64 btrfs_file_aio_write problem? Date: Fri, 15 Aug 2014 12:25:36 +0200 Message-ID: <2844720.iC9EVhHRRD@merkaba> In-Reply-To: <2628463.rfvAuWT57X@workstation> References: <2553353.cJGYBglVXM@stakan> <2168796.1W5hd7vOnH@stakan> <2628463.rfvAuWT57X@workstation> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Sender: linux-btrfs-owner@vger.kernel.org List-ID: Hi Valdis, Am Freitag, 15. August 2014, 14:10:37 schrieb Valdis Voronin: > System freeze after IO chaos. See attached image. > > В письме от 13 августа 2014 20:40:49 пользователь Valdis Voronin написал: > > Sorry subject should be btrfs 3.14-2-amd64 btrfs_file_aio_write problem. > > > > On Wednesday, August 13, 2014 08:39:15 PM you wrote: > > > Hello, > > > > > > I am using here Debian Jessie whith classic Raid1 mdadm, with two ssd > > > devices (Crucial_CT256MX100SSD1) and btrfs 3.14-2-amd64 #1 SMP Debian > > > 3.14.13-2 (2014-07-24) x86_64 GNU/Linux. > > > The server under load. After some days or a week is see in messages (log > > > files, see below.). > > > The server is not freeze but works not correctly. There are no > > > possibility > > > to reboot. > > > Is this a bug? May be somebody has an ider for mount options or kernel > > > upgrade? > > > > > > cat > fstab > > > > > > /dev/md/2 / btrfs defaults 0 0 Please include outputs of btrfs fi sh / btrfs fi df / Also I do recommend against using SoftRAID on SSDs until SoftRAID actually uses ATA TRIM to sync empty areas of both devices to one another. I am not sure whether it does so in 3.14. Otherwise lets see what BTRFS developers have to say about this backtrace. Since it happens with 3.14 it does not seem to be related with the hangs after switching BTRFS to workqueues which happen in 3.15 and 3.16. There are two patches related to fixing this issue, but the fix may still be incomplete. Thanks, Martin > > > Regards > > > Vladislav > > > > > > cat > messages > > > > > > Aug 12 06:25:45 srv1 rsyslogd: [origin software="rsyslogd" > > > swVersion="7.6.3" x-pid="1676" x-info="http://www.rsyslog.com"] rsyslogd > > > was HUPed > > > Aug 12 06:28:36 srv1 rsyslogd-2359: action 'action 17' resumed (module > > > 'builtin:ompipe') [try http://www.rsyslog.com/e/2359 ] > > > Aug 12 07:02:04 srv1 rsyslogd-2007: action 'action 17' suspended, next > > > retry is Tue Aug 12 07:02:34 2014 [try http://www.rsyslog.com/e/2007 ] > > > Aug 12 09:03:41 srv1 kernel: [163223.598939] mysqld D > > > ffff88040b2a2e38 0 11941 4011 0x00000000 > > > Aug 12 09:03:41 srv1 kernel: [163223.598941] ffff88040b2a2a20 > > > 0000000000000086 0000000000014380 ffff880428eeffd8 > > > Aug 12 09:03:41 srv1 kernel: [163223.598943] 0000000000014380 > > > ffff88040b2a2a20 ffff88081ea54c10 ffff88081edc7608 > > > Aug 12 09:03:41 srv1 kernel: [163223.598944] 0000000000000002 > > > ffffffff81121000 ffff880428eef6d0 ffff880428eef7b0 > > > Aug 12 09:03:41 srv1 kernel: [163223.598946] Call Trace: > > > Aug 12 09:03:41 srv1 kernel: [163223.598950] [] ? > > > wait_on_page_read+0x60/0x60 > > > Aug 12 09:03:41 srv1 kernel: [163223.598954] [] ? > > > io_schedule+0x94/0x130 > > > Aug 12 09:03:41 srv1 kernel: [163223.598955] [] ? > > > sleep_on_page+0x5/0x10 > > > Aug 12 09:03:41 srv1 kernel: [163223.598956] [] ? > > > __wait_on_bit+0x54/0x80 > > > Aug 12 09:03:41 srv1 kernel: [163223.598960] [] ? > > > autoremove_wake_function+0x30/0x30 > > > Aug 12 09:03:41 srv1 kernel: [163223.598965] [] ? > > > shrink_inactive_list+0x187/0x4d0 > > > Aug 12 09:03:41 srv1 kernel: [163223.598968] [] ? > > > shrink_zone+0x5e/0x180 > > > Aug 12 09:03:41 srv1 kernel: [163223.598971] [] ? > > > try_to_free_mem_cgroup_pages+0xc5/0x150 > > > Aug 12 09:03:41 srv1 kernel: [163223.598975] [] ? > > > __mem_cgroup_try_charge+0x55f/0x6a0 > > > Aug 12 09:03:41 srv1 kernel: [163223.598990] [] ? > > > mem_cgroup_charge_common+0x3d/0x90 > > > Aug 12 09:03:41 srv1 kernel: [163223.598995] [] ? > > > add_to_page_cache_lru+0x11/0x40 > > > Aug 12 09:03:41 srv1 kernel: [163223.599002] [] ? > > > prepare_pages.isra.19+0xae/0x170 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599012] [] ? > > > btrfs_file_aio_write+0x215/0x550 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599015] [] ? > > > do_sync_write+0x57/0x90 > > > Aug 12 09:03:41 srv1 kernel: [163223.599018] [] ? > > > SyS_write+0x3d/0xa0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599054] ffff8807f6a1b1f0 > > > 0000000000000086 0000000000014380 ffff88053c2c3fd8 > > > Aug 12 09:03:41 srv1 kernel: [163223.599057] 0000000000000002 > > > ffffffff81121000 ffff88053c2c3650 ffff88053c2c3730 > > > Aug 12 09:03:41 srv1 kernel: [163223.599059] [] ? > > > wait_on_page_read+0x60/0x60 > > > Aug 12 09:03:41 srv1 kernel: [163223.599062] [] ? > > > sleep_on_page+0x5/0x10 > > > Aug 12 09:03:41 srv1 kernel: [163223.599064] [] ? > > > wait_on_page_bit+0x7f/0x90 > > > Aug 12 09:03:41 srv1 kernel: [163223.599067] [] ? > > > shrink_page_list+0x667/0xa50 > > > Aug 12 09:03:41 srv1 kernel: [163223.599069] [] ? > > > shrink_lruvec+0x2e9/0x600 > > > Aug 12 09:03:41 srv1 kernel: [163223.599072] [] ? > > > do_try_to_free_pages+0xe0/0x550 > > > Aug 12 09:03:41 srv1 kernel: [163223.599076] [] ? > > > try_to_free_mem_cgroup_pages+0xc5/0x150 > > > Aug 12 09:03:41 srv1 kernel: [163223.599079] [] ? > > > __mem_cgroup_try_charge+0x55f/0x6a0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599082] [] ? > > > mem_cgroup_charge_common+0x3d/0x90 > > > Aug 12 09:03:41 srv1 kernel: [163223.599084] [] ? > > > add_to_page_cache_locked+0x2f/0x150 > > > Aug 12 09:03:41 srv1 kernel: [163223.599091] [] ? > > > extent_readpages+0xb7/0x190 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599098] [] ? > > > __do_page_cache_readahead+0x193/0x240 > > > Aug 12 09:03:41 srv1 kernel: [163223.599101] [] ? > > > generic_file_aio_read+0x459/0x6d0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599103] [] ? > > > vfs_read+0x91/0x160 > > > Aug 12 09:03:41 srv1 kernel: [163223.599106] [] ? > > > system_call_fastpath+0x16/0x1b > > > Aug 12 09:03:41 srv1 kernel: [163223.599168] kworker/u16:2 D > > > ffff880730ea0868 0 27951 2 0x00000000 > > > Aug 12 09:03:41 srv1 kernel: [163223.599172] ffff880730ea0450 > > > 0000000000000046 0000000000014380 ffff88000721ffd8 > > > Aug 12 09:03:41 srv1 kernel: [163223.599174] ffff88000721f880 > > > 0000000000000002 ffffffff81121000 ffff8805827e3d98 > > > Aug 12 09:03:41 srv1 kernel: [163223.599176] [] ? > > > wait_on_page_read+0x60/0x60 > > > Aug 12 09:03:41 srv1 kernel: [163223.599179] [] ? > > > sleep_on_page+0x5/0x10 > > > Aug 12 09:03:41 srv1 kernel: [163223.599181] [] ? > > > __lock_page+0x65/0x70 > > > Aug 12 09:03:41 srv1 kernel: [163223.599189] [] ? > > > lock_delalloc_pages+0x10d/0x190 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599199] [] ? > > > submit_extent_page.isra.36+0x1af/0x230 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599209] [] ? > > > end_extent_writepage+0x90/0x90 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599215] [] ? > > > extent_write_cache_pages.isra.29.constprop.49+0x207/0x340 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599225] [] ? > > > btrfs_submit_direct+0x6b0/0x6b0 [btrfs] > > > Aug 12 09:03:41 srv1 kernel: [163223.599229] [] ? > > > writeback_sb_inodes+0x19c/0x3d0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599232] [] ? > > > wb_writeback+0x243/0x2d0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599236] [] ? > > > __switch_to+0x11b/0x4b0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599240] [] ? > > > worker_thread+0x116/0x3b0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599243] [] ? > > > kthread+0xb8/0xd0 > > > Aug 12 09:03:41 srv1 kernel: [163223.599245] [] ? > > > ret_from_fork+0x7c/0xb0 > > > Aug 12 09:05:41 srv1 kernel: [163343.527812] Call Trace: > > > > > > Grus > > > Valdis > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7