From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com ([209.132.183.28]:41149 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758968AbcCDVDo (ORCPT ); Fri, 4 Mar 2016 16:03:44 -0500 Date: Fri, 4 Mar 2016 16:03:42 -0500 From: Brian Foster To: Stefan Priebe Cc: linux-fsdevel@vger.kernel.org, "xfs-masters@oss.sgi.com" , "xfs@oss.sgi.com" Subject: Re: xfs trace in 4.4.2 / also in 4.3.3 WARNING fs/xfs/xfs_aops.c:1232 xfs_vm_releasepage Message-ID: <20160304210341.GA8035@bfoster.bfoster> References: <56C81D94.7090603@profihost.ag> <20160220144533.GA36182@bfoster.bfoster> <56D9D834.2000303@profihost.ag> <20160304191329.GC3758@bfoster.bfoster> <56D9E9BE.40101@profihost.ag> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56D9E9BE.40101@profihost.ag> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Fri, Mar 04, 2016 at 09:02:06PM +0100, Stefan Priebe wrote: > > Am 04.03.2016 um 20:13 schrieb Brian Foster: > >On Fri, Mar 04, 2016 at 07:47:16PM +0100, Stefan Priebe wrote: > >>Am 20.02.2016 um 19:02 schrieb Stefan Priebe - Profihost AG: > >>> > >>>>Am 20.02.2016 um 15:45 schrieb Brian Foster : > >>>> > >>>>>On Sat, Feb 20, 2016 at 09:02:28AM +0100, Stefan Priebe wrote: > >>>>>Hi, > >>>>> > >>>>>got this one today. Not sure if this is a bug. > >>>> > >>>>That looks like the releasepage() delayed allocation block warning. I'm > >>>>not sure we've had any fixes for (or reports of) that issue since the > >>>>v4.2 timeframe. > >>>> > >>>>What is the xfs_info of the associated filesystem? Also, do you have any > >>>>insight as to the possible reproducer application or workload? Is this > >>>>reproducible at all? Note that this is a WARN_ON_ONCE(), so the warning > >>>>won't fire again regardless until after a reboot. > >> > >>Toda i got this one running 4.3.3. > >> > >>[154152.949610] ------------[ cut here ]------------ > >>[154152.950704] WARNING: CPU: 0 PID: 79 at fs/xfs/xfs_aops.c:1232 > >>xfs_vm_releasepage+0xc3/0xf0() > >>[154152.952596] Modules linked in: netconsole mpt3sas raid_class > >>nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp ipt_REJECT > >>nf_reject_ipv4 xt_owner xt_multiport iptable_filter ip_tables x_tables 8021q > >>garp coretemp k8temp ehci_pci ehci_hcd sb_edac ipmi_si usbcore edac_core > >>ipmi_msghandler i2c_i801 usb_common button btrfs xor raid6_pq sg igb sd_mod > >>i2c_algo_bit isci i2c_core libsas ahci ptp libahci scsi_transport_sas > >>megaraid_sas pps_core > >>[154152.963240] CPU: 0 PID: 79 Comm: kswapd0 Not tainted 4.4.3+3-ph #1 > >>[154152.964625] Hardware name: Supermicro > >>X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 1.0a > >>03/06/2012 > >>[154152.967029] 0000000000000000 ffff88103dd67a98 ffffffffa73c3b5f > >>0000000000000000 > >>[154152.968836] ffffffffa7a5063b ffff88103dd67ad8 ffffffffa7083757 > >>0000000000000000 > >>[154152.970641] 0000000000000001 ffffea0001e7bfc0 ffff88071ef72dd0 > >>ffffea0001e7bfe0 > >>[154152.972447] Call Trace: > >>[154152.973011] [] dump_stack+0x63/0x84 > >>[154152.974167] [] warn_slowpath_common+0x97/0xe0 > >>[154152.975515] [] warn_slowpath_null+0x1a/0x20 > >>[154152.976826] [] xfs_vm_releasepage+0xc3/0xf0 > >>[154152.978137] [] try_to_release_page+0x32/0x50 > >>[154152.979467] [] shrink_active_list+0x3ce/0x3e0 > >>[154152.980816] [] shrink_lruvec+0x687/0x7d0 > >>[154152.982068] [] shrink_zone+0xdc/0x2c0 > >>[154152.983262] [] kswapd+0x4f9/0x970 > >>[154152.984380] [] ? > >>mem_cgroup_shrink_node_zone+0x1a0/0x1a0 > >>[154152.985942] [] kthread+0xc9/0xe0 > >>[154152.987040] [] ? kthread_stop+0x100/0x100 > >>[154152.988313] [] ret_from_fork+0x3f/0x70 > >>[154152.989527] [] ? kthread_stop+0x100/0x100 > >>[154152.990818] ---[ end trace 3fac2515e92c7cb1 ]--- > >> > >>This time with an xfs info: > >># xfs_info / > >>meta-data=/dev/disk/by-uuid/9befe321-e9cc-4e31-82df-efabb3211bac isize=256 > >>agcount=4, agsize=58224256 blks > >> = sectsz=512 attr=2, projid32bit=0 > >> = crc=0 finobt=0 > >>data = bsize=4096 blocks=232897024, imaxpct=25 > >> = sunit=64 swidth=384 blks > >>naming =version 2 bsize=4096 ascii-ci=0 ftype=0 > >>log =internal bsize=4096 blocks=113728, version=2 > >> = sectsz=512 sunit=64 blks, lazy-count=1 > >>realtime =none extsz=4096 blocks=0, rtextents=0 > >> > > > >Can you describe the workload to the filesystem? > > At the time of this trace the rsync backup of the fs has started. So the > workload was going from nearly idle to 4000 iop/s read at 60 MB/s peak. > Interesting. The warning is associated with releasing a page that has a delayed allocation when it shouldn't. That means something had written to a file to cause the delalloc in the first place. Any idea what could have been writing at the time or shortly before the rsync read workload had kicked in? Brian > Stefan > > >Brian > > > >>> > >>>> > >>>>Brian > >>>> > >>>>>[67674.907736] ------------[ cut here ]------------ > >>>>>[67674.955858] WARNING: CPU: 5 PID: 197 at fs/xfs/xfs_aops.c:1232 > >>>>>xfs_vm_releasepage+0xa9/0xe0() > >>>>>[67675.005345] Modules linked in: dm_mod netconsole ipt_REJECT > >>>>>nf_reject_ipv4 mpt3sas raid_class xt_multiport iptable_filter ip_tabl > >>>>>es x_tables 8021q garp bonding coretemp loop usbhid ehci_pci ehci_hcd > >>>>>sb_edac ipmi_si usbcore i2c_i801 edac_core usb_common ipmi_msg > >>>>>handler button btrfs xor raid6_pq raid1 md_mod sg igb sd_mod i2c_algo_bit > >>>>>ixgbe ahci i2c_core mdio isci libahci libsas ptp megaraid_ > >>>>>sas scsi_transport_sas pps_core > >>>>>[67675.221939] CPU: 5 PID: 197 Comm: kswapd0 Not tainted 4.4.2+1-ph #1 > >>>>>[67675.277120] Hardware name: Supermicro > >>>>>X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 > >>>>>[67675.335176] ffffffffa3a5065d ffff88007950fa98 ffffffffa33bd4e1 > >>>>>0000000000000001 > >>>>>[67675.392983] 0000000000000000 ffff88007950fad8 ffffffffa3083587 > >>>>>ffff88007950fae8 > >>>>>[67675.449743] 0000000000000001 ffffea0020883480 ffff880cf4b9cdd0 > >>>>>ffffea00208834a0 > >>>>>[67675.506112] Call Trace: > >>>>>[67675.561285] [] dump_stack+0x45/0x64 > >>>>>[67675.619364] [] warn_slowpath_common+0x97/0xe0 > >>>>>[67675.675719] [] warn_slowpath_null+0x1a/0x20 > >>>>>[67675.731113] [] xfs_vm_releasepage+0xa9/0xe0 > >>>>>[67675.786116] [] ? page_mkclean_one+0xd0/0xd0 > >>>>>[67675.844216] [] ? anon_vma_prepare+0x150/0x150 > >>>>>[67675.903862] [] try_to_release_page+0x32/0x50 > >>>>>[67675.957625] [] shrink_active_list+0x3ce/0x3e0 > >>>>>[67676.011497] [] shrink_lruvec+0x687/0x7d0 > >>>>>[67676.064980] [] shrink_zone+0xdc/0x2c0 > >>>>>[67676.118828] [] kswapd+0x4f9/0x930 > >>>>>[67676.172075] [] ? > >>>>>mem_cgroup_shrink_node_zone+0x150/0x150 > >>>>>[67676.225139] [] kthread+0xc9/0xe0 > >>>>>[67676.277539] [] ? kthread_stop+0xe0/0xe0 > >>>>>[67676.330124] [] ret_from_fork+0x3f/0x70 > >>>>>[67676.381816] [] ? kthread_stop+0xe0/0xe0 > >>>>>[67676.433499] ---[ end trace cb1827fe308f7f6b ]--- > >>>>> > >>>>>Greets Stefan > >>>>> > >>>>>_______________________________________________ > >>>>>xfs mailing list > >>>>>xfs@oss.sgi.com > >>>>>http://oss.sgi.com/mailman/listinfo/xfs > >> > >>_______________________________________________ > >>xfs mailing list > >>xfs@oss.sgi.com > >>http://oss.sgi.com/mailman/listinfo/xfs