* rm Tainted warning after kernel update. @ 2015-09-14 18:57 Grant Keller 2015-09-15 11:00 ` Brian Foster 0 siblings, 1 reply; 7+ messages in thread From: Grant Keller @ 2015-09-14 18:57 UTC (permalink / raw) To: xfs Hello, I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs: Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed. -- Grant Keller System Operations 707-237-2451 grant.keller@sonic.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-14 18:57 rm Tainted warning after kernel update Grant Keller @ 2015-09-15 11:00 ` Brian Foster 2015-09-15 22:32 ` Grant Keller 0 siblings, 1 reply; 7+ messages in thread From: Brian Foster @ 2015-09-15 11:00 UTC (permalink / raw) To: Grant Keller; +Cc: xfs On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: > Hello, > > I have a server running Scientific Linux 6.7, and since updating to > kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing > in our message logs: > > Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ > Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 > d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) > Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 > Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl > auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib > nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 > nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm > iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc > parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe > ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb > ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi > ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core > dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] > Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G > W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 > Sep 14 11:43:03 localhost kernel: Call Trace: > Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? > warn_slowpath_common+0x91/0xe0 > Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? > warn_slowpath_null+0x1a/0x20 > Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? > d_delete+0x260/0x2c0 > Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 > Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? > do_rmdir+0x184/0x1f0 > Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 > Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? > audit_syscall_entry+0x1d7/0x200 > Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? > sys_unlinkat+0x2d/0x40 > Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? > system_call_fastpath+0x16/0x1b > Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- > > This happens when we are expiring older backups from the archives, so I > have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 > installed. Looking for advice on how to proceed. > This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before. I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal? Also, what kernel were you running before this started to occur? Brian > -- > Grant Keller > System Operations > 707-237-2451 > grant.keller@sonic.com > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-15 11:00 ` Brian Foster @ 2015-09-15 22:32 ` Grant Keller 2015-09-16 10:50 ` Brian Foster 0 siblings, 1 reply; 7+ messages in thread From: Grant Keller @ 2015-09-15 22:32 UTC (permalink / raw) To: Brian Foster; +Cc: xfs On 09/15/2015 04:00 AM, Brian Foster wrote: > On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: >> Hello, >> >> I have a server running Scientific Linux 6.7, and since updating to >> kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing >> in our message logs: >> >> Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ >> Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 >> d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) >> Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 >> Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl >> auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib >> nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 >> nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT >> nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter >> ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm >> iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc >> parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe >> ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb >> ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi >> ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core >> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] >> Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G >> W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 >> Sep 14 11:43:03 localhost kernel: Call Trace: >> Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? >> warn_slowpath_common+0x91/0xe0 >> Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? >> warn_slowpath_null+0x1a/0x20 >> Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? >> d_delete+0x260/0x2c0 >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? >> do_rmdir+0x184/0x1f0 >> Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 >> Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? >> audit_syscall_entry+0x1d7/0x200 >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? >> sys_unlinkat+0x2d/0x40 >> Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? >> system_call_fastpath+0x16/0x1b >> Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- >> >> This happens when we are expiring older backups from the archives, so I >> have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 >> installed. Looking for advice on how to proceed. >> > This looks like something funky going on in the vfs. The warning is from > unhash_offsprings() and it appears to be complaining about a refcount on > a dentry that is a child of a directory being removed. It checks a > refcount on a dentry in one loop and either drops it or moves it to > another list for apparent deletion. The second iteration of the > aforementioned list sees a refcount on an object that wasn't there > before. > > I suspect this means something is going from 0->1 unexpectedly, but I'm > not familiar enough with that code to grok why that shouldn't happen and > how it could without reproducing it and digging into it from there. Have > you identified an explicit reproducer? I assume files are simply being > removed with 'rm -rf' here..? If so, does anything else have access to > this directory structure (e.g., separate commands, a running backup > application?) at the the time of removal. There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings. > > Also, what kernel were you running before this started to occur? 2.6.32-573.el6.x86_64 was the previous kernel. > > Brian > > -- Grant Keller System Operations 707-237-2451 grant.keller@sonic.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-15 22:32 ` Grant Keller @ 2015-09-16 10:50 ` Brian Foster 2015-09-16 11:13 ` Lukáš Czerner 0 siblings, 1 reply; 7+ messages in thread From: Brian Foster @ 2015-09-16 10:50 UTC (permalink / raw) To: Grant Keller; +Cc: lczerner, xfs cc Lukas On Tue, Sep 15, 2015 at 03:32:22PM -0700, Grant Keller wrote: > On 09/15/2015 04:00 AM, Brian Foster wrote: > > On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: > >> Hello, > >> > >> I have a server running Scientific Linux 6.7, and since updating to > >> kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing > >> in our message logs: > >> > >> Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ > >> Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 > >> d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) > >> Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 > >> Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl > >> auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib > >> nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 > >> nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT > >> nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > >> ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm > >> iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc > >> parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe > >> ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb > >> ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi > >> ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core > >> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] > >> Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G > >> W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 > >> Sep 14 11:43:03 localhost kernel: Call Trace: > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? > >> warn_slowpath_common+0x91/0xe0 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? > >> warn_slowpath_null+0x1a/0x20 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? > >> d_delete+0x260/0x2c0 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? > >> do_rmdir+0x184/0x1f0 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? > >> audit_syscall_entry+0x1d7/0x200 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? > >> sys_unlinkat+0x2d/0x40 > >> Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? > >> system_call_fastpath+0x16/0x1b > >> Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- > >> > >> This happens when we are expiring older backups from the archives, so I > >> have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 > >> installed. Looking for advice on how to proceed. > >> > > This looks like something funky going on in the vfs. The warning is from > > unhash_offsprings() and it appears to be complaining about a refcount on > > a dentry that is a child of a directory being removed. It checks a > > refcount on a dentry in one loop and either drops it or moves it to > > another list for apparent deletion. The second iteration of the > > aforementioned list sees a refcount on an object that wasn't there > > before. > > > > I suspect this means something is going from 0->1 unexpectedly, but I'm > > not familiar enough with that code to grok why that shouldn't happen and > > how it could without reproducing it and digging into it from there. Have > > you identified an explicit reproducer? I assume files are simply being > > removed with 'rm -rf' here..? If so, does anything else have access to > > this directory structure (e.g., separate commands, a running backup > > application?) at the the time of removal. > There could be something else running, but I would have to investigate > the next time this happens. The rm -rf is called by our backup program > expiring older backups from the filesystem. The thing is, the > expirations happen on a nightly basis, but we don't always see these > warnings in the logs. On the nights we do, there are 1000+ warnings. > > > > Also, what kernel were you running before this started to occur? > 2.6.32-573.el6.x86_64 was the previous kernel. Interesting... there are only a few fs changes between this kernel and the current. One of them is this: 959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir ... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code. FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved. Brian > > > > Brian > > > > > > -- > Grant Keller > System Operations > 707-237-2451 > grant.keller@sonic.com > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-16 10:50 ` Brian Foster @ 2015-09-16 11:13 ` Lukáš Czerner 2015-09-16 11:42 ` Brian Foster 0 siblings, 1 reply; 7+ messages in thread From: Lukáš Czerner @ 2015-09-16 11:13 UTC (permalink / raw) To: Brian Foster; +Cc: Grant Keller, xfs On Wed, 16 Sep 2015, Brian Foster wrote: > Date: Wed, 16 Sep 2015 06:50:23 -0400 > From: Brian Foster <bfoster@redhat.com> > To: Grant Keller <grant.keller@sonic.com> > Cc: lczerner@redhat.com, xfs@oss.sgi.com > Subject: Re: rm Tainted warning after kernel update. > > cc Lukas > > On Tue, Sep 15, 2015 at 03:32:22PM -0700, Grant Keller wrote: > > On 09/15/2015 04:00 AM, Brian Foster wrote: > > > On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: > > >> Hello, > > >> > > >> I have a server running Scientific Linux 6.7, and since updating to > > >> kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing > > >> in our message logs: > > >> > > >> Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ > > >> Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 > > >> d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) > > >> Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 > > >> Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl > > >> auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib > > >> nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 > > >> nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT > > >> nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > > >> ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm > > >> iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc > > >> parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe > > >> ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb > > >> ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi > > >> ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core > > >> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] > > >> Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G > > >> W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 > > >> Sep 14 11:43:03 localhost kernel: Call Trace: > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? > > >> warn_slowpath_common+0x91/0xe0 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? > > >> warn_slowpath_null+0x1a/0x20 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? > > >> d_delete+0x260/0x2c0 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? > > >> do_rmdir+0x184/0x1f0 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? > > >> audit_syscall_entry+0x1d7/0x200 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? > > >> sys_unlinkat+0x2d/0x40 > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? > > >> system_call_fastpath+0x16/0x1b > > >> Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- > > >> > > >> This happens when we are expiring older backups from the archives, so I > > >> have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 > > >> installed. Looking for advice on how to proceed. > > >> > > > This looks like something funky going on in the vfs. The warning is from > > > unhash_offsprings() and it appears to be complaining about a refcount on > > > a dentry that is a child of a directory being removed. It checks a > > > refcount on a dentry in one loop and either drops it or moves it to > > > another list for apparent deletion. The second iteration of the > > > aforementioned list sees a refcount on an object that wasn't there > > > before. > > > > > > I suspect this means something is going from 0->1 unexpectedly, but I'm > > > not familiar enough with that code to grok why that shouldn't happen and > > > how it could without reproducing it and digging into it from there. Have > > > you identified an explicit reproducer? I assume files are simply being > > > removed with 'rm -rf' here..? If so, does anything else have access to > > > this directory structure (e.g., separate commands, a running backup > > > application?) at the the time of removal. > > There could be something else running, but I would have to investigate > > the next time this happens. The rm -rf is called by our backup program > > expiring older backups from the filesystem. The thing is, the > > expirations happen on a nightly basis, but we don't always see these > > warnings in the logs. On the nights we do, there are 1000+ warnings. > > > > > > Also, what kernel were you running before this started to occur? > > 2.6.32-573.el6.x86_64 was the previous kernel. > > Interesting... there are only a few fs changes between this kernel and > the current. One of them is this: > > 959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir > > ... which actually introduces the unhash_offsprings() thing. I've cc'd > Lukas who is probably more familiar with this code. > > FWIW, I suspect the more you can elaborate on what the backup > application might be doing here (beyond just the rm -rf), the more > likely this can be reproduced and resolved. Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that. -Lukas > > Brian > > > > > > > Brian > > > > > > > > > > -- > > Grant Keller > > System Operations > > 707-237-2451 > > grant.keller@sonic.com > > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-16 11:13 ` Lukáš Czerner @ 2015-09-16 11:42 ` Brian Foster 2015-09-16 12:21 ` Lukáš Czerner 0 siblings, 1 reply; 7+ messages in thread From: Brian Foster @ 2015-09-16 11:42 UTC (permalink / raw) To: Lukáš Czerner; +Cc: Grant Keller, xfs On Wed, Sep 16, 2015 at 01:13:20PM +0200, Lukáš Czerner wrote: > On Wed, 16 Sep 2015, Brian Foster wrote: > > > Date: Wed, 16 Sep 2015 06:50:23 -0400 > > From: Brian Foster <bfoster@redhat.com> > > To: Grant Keller <grant.keller@sonic.com> > > Cc: lczerner@redhat.com, xfs@oss.sgi.com > > Subject: Re: rm Tainted warning after kernel update. > > > > cc Lukas > > > > On Tue, Sep 15, 2015 at 03:32:22PM -0700, Grant Keller wrote: > > > On 09/15/2015 04:00 AM, Brian Foster wrote: > > > > On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: > > > >> Hello, > > > >> > > > >> I have a server running Scientific Linux 6.7, and since updating to > > > >> kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing > > > >> in our message logs: > > > >> > > > >> Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ > > > >> Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 > > > >> d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) > > > >> Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 > > > >> Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl > > > >> auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib > > > >> nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 > > > >> nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT > > > >> nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > > > >> ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm > > > >> iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc > > > >> parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe > > > >> ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb > > > >> ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi > > > >> ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core > > > >> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] > > > >> Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G > > > >> W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 > > > >> Sep 14 11:43:03 localhost kernel: Call Trace: > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? > > > >> warn_slowpath_common+0x91/0xe0 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? > > > >> warn_slowpath_null+0x1a/0x20 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? > > > >> d_delete+0x260/0x2c0 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? > > > >> do_rmdir+0x184/0x1f0 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? > > > >> audit_syscall_entry+0x1d7/0x200 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? > > > >> sys_unlinkat+0x2d/0x40 > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? > > > >> system_call_fastpath+0x16/0x1b > > > >> Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- > > > >> > > > >> This happens when we are expiring older backups from the archives, so I > > > >> have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 > > > >> installed. Looking for advice on how to proceed. > > > >> > > > > This looks like something funky going on in the vfs. The warning is from > > > > unhash_offsprings() and it appears to be complaining about a refcount on > > > > a dentry that is a child of a directory being removed. It checks a > > > > refcount on a dentry in one loop and either drops it or moves it to > > > > another list for apparent deletion. The second iteration of the > > > > aforementioned list sees a refcount on an object that wasn't there > > > > before. > > > > > > > > I suspect this means something is going from 0->1 unexpectedly, but I'm > > > > not familiar enough with that code to grok why that shouldn't happen and > > > > how it could without reproducing it and digging into it from there. Have > > > > you identified an explicit reproducer? I assume files are simply being > > > > removed with 'rm -rf' here..? If so, does anything else have access to > > > > this directory structure (e.g., separate commands, a running backup > > > > application?) at the the time of removal. > > > There could be something else running, but I would have to investigate > > > the next time this happens. The rm -rf is called by our backup program > > > expiring older backups from the filesystem. The thing is, the > > > expirations happen on a nightly basis, but we don't always see these > > > warnings in the logs. On the nights we do, there are 1000+ warnings. > > > > > > > > Also, what kernel were you running before this started to occur? > > > 2.6.32-573.el6.x86_64 was the previous kernel. > > > > Interesting... there are only a few fs changes between this kernel and > > the current. One of them is this: > > > > 959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir > > > > ... which actually introduces the unhash_offsprings() thing. I've cc'd > > Lukas who is probably more familiar with this code. > > > > FWIW, I suspect the more you can elaborate on what the backup > > application might be doing here (beyond just the rm -rf), the more > > likely this can be reproduced and resolved. > > Hi the lockdep warning should be fixed in the recent rhel6 kernel. > Sorry about that. > My understanding is that this patch has been reverted... Grant, If a newer kernel is not yet available, you might want to revert to the previous version until one is and then go straight to that. Lukas, If the patch is indeed reverted, then clearly this problem will go away. That aside, where does lockdep come into play? Note the warning reported above: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() ... is an explicit WARN_ON() in the code added by this commit. Are we referring to the same issue here? Brian > -Lukas > > > > > Brian > > > > > > > > > > Brian > > > > > > > > > > > > > > -- > > > Grant Keller > > > System Operations > > > 707-237-2451 > > > grant.keller@sonic.com > > > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs > > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rm Tainted warning after kernel update. 2015-09-16 11:42 ` Brian Foster @ 2015-09-16 12:21 ` Lukáš Czerner 0 siblings, 0 replies; 7+ messages in thread From: Lukáš Czerner @ 2015-09-16 12:21 UTC (permalink / raw) To: Brian Foster; +Cc: Grant Keller, xfs [-- Attachment #1: Type: TEXT/PLAIN, Size: 6819 bytes --] On Wed, 16 Sep 2015, Brian Foster wrote: > Date: Wed, 16 Sep 2015 07:42:31 -0400 > From: Brian Foster <bfoster@redhat.com> > To: Lukáš Czerner <lczerner@redhat.com> > Cc: Grant Keller <grant.keller@sonic.com>, xfs@oss.sgi.com > Subject: Re: rm Tainted warning after kernel update. > > On Wed, Sep 16, 2015 at 01:13:20PM +0200, Lukáš Czerner wrote: > > On Wed, 16 Sep 2015, Brian Foster wrote: > > > > > Date: Wed, 16 Sep 2015 06:50:23 -0400 > > > From: Brian Foster <bfoster@redhat.com> > > > To: Grant Keller <grant.keller@sonic.com> > > > Cc: lczerner@redhat.com, xfs@oss.sgi.com > > > Subject: Re: rm Tainted warning after kernel update. > > > > > > cc Lukas > > > > > > On Tue, Sep 15, 2015 at 03:32:22PM -0700, Grant Keller wrote: > > > > On 09/15/2015 04:00 AM, Brian Foster wrote: > > > > > On Mon, Sep 14, 2015 at 11:57:37AM -0700, Grant Keller wrote: > > > > >> Hello, > > > > >> > > > > >> I have a server running Scientific Linux 6.7, and since updating to > > > > >> kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing > > > > >> in our message logs: > > > > >> > > > > >> Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ > > > > >> Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 > > > > >> d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) > > > > >> Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 > > > > >> Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl > > > > >> auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib > > > > >> nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 > > > > >> nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT > > > > >> nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter > > > > >> ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm > > > > >> iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc > > > > >> parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe > > > > >> ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb > > > > >> ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi > > > > >> ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core > > > > >> dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] > > > > >> Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G > > > > >> W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 > > > > >> Sep 14 11:43:03 localhost kernel: Call Trace: > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? > > > > >> warn_slowpath_common+0x91/0xe0 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? > > > > >> warn_slowpath_null+0x1a/0x20 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? > > > > >> d_delete+0x260/0x2c0 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? > > > > >> do_rmdir+0x184/0x1f0 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? > > > > >> audit_syscall_entry+0x1d7/0x200 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? > > > > >> sys_unlinkat+0x2d/0x40 > > > > >> Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? > > > > >> system_call_fastpath+0x16/0x1b > > > > >> Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]--- > > > > >> > > > > >> This happens when we are expiring older backups from the archives, so I > > > > >> have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 > > > > >> installed. Looking for advice on how to proceed. > > > > >> > > > > > This looks like something funky going on in the vfs. The warning is from > > > > > unhash_offsprings() and it appears to be complaining about a refcount on > > > > > a dentry that is a child of a directory being removed. It checks a > > > > > refcount on a dentry in one loop and either drops it or moves it to > > > > > another list for apparent deletion. The second iteration of the > > > > > aforementioned list sees a refcount on an object that wasn't there > > > > > before. > > > > > > > > > > I suspect this means something is going from 0->1 unexpectedly, but I'm > > > > > not familiar enough with that code to grok why that shouldn't happen and > > > > > how it could without reproducing it and digging into it from there. Have > > > > > you identified an explicit reproducer? I assume files are simply being > > > > > removed with 'rm -rf' here..? If so, does anything else have access to > > > > > this directory structure (e.g., separate commands, a running backup > > > > > application?) at the the time of removal. > > > > There could be something else running, but I would have to investigate > > > > the next time this happens. The rm -rf is called by our backup program > > > > expiring older backups from the filesystem. The thing is, the > > > > expirations happen on a nightly basis, but we don't always see these > > > > warnings in the logs. On the nights we do, there are 1000+ warnings. > > > > > > > > > > Also, what kernel were you running before this started to occur? > > > > 2.6.32-573.el6.x86_64 was the previous kernel. > > > > > > Interesting... there are only a few fs changes between this kernel and > > > the current. One of them is this: > > > > > > 959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir > > > > > > ... which actually introduces the unhash_offsprings() thing. I've cc'd > > > Lukas who is probably more familiar with this code. > > > > > > FWIW, I suspect the more you can elaborate on what the backup > > > application might be doing here (beyond just the rm -rf), the more > > > likely this can be reproduced and resolved. > > > > Hi the lockdep warning should be fixed in the recent rhel6 kernel. > > Sorry about that. > > > > My understanding is that this patch has been reverted... > > Grant, > > If a newer kernel is not yet available, you might want to revert to the > previous version until one is and then go straight to that. > > Lukas, > > If the patch is indeed reverted, then clearly this problem will go away. > That aside, where does lockdep come into play? Note the warning reported > above: > > WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() > > ... is an explicit WARN_ON() in the code added by this commit. Are we > referring to the same issue here? > > Brian Hi Brian, seems like we're not referring to the same issue as I though we're talking about the lockdep problem, but apparently we're not. Can we get a RH bugzilla for this issue so it can be dealt with properly ? Thanks! -Lukas [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-09-16 12:22 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-09-14 18:57 rm Tainted warning after kernel update Grant Keller 2015-09-15 11:00 ` Brian Foster 2015-09-15 22:32 ` Grant Keller 2015-09-16 10:50 ` Brian Foster 2015-09-16 11:13 ` Lukáš Czerner 2015-09-16 11:42 ` Brian Foster 2015-09-16 12:21 ` Lukáš Czerner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox