Delay in folder access

Delay in folder access

[Daniel L. Miller]

I have an XFS partition on simple 4-disk RAID-10 array.  The array is 
mounted at /raid.  I have a variety of subfolders - one of which is 
/raid/data - which in turn has a number of subfolders.

Doing a simple "ls /raid" has no problems.  Similarly, no issues with 
"ls /raid/data/stuff".  But ... executing "ls /raid/data" results in a 
delay, sometimes in minutes, before it returns.

I don't see any errors in dmesg or syslog.  Does this indicate a pending 
crash?

Also, I do have a ".fsr" folder under /raid.  All the folders listed, 
which appear empty, are dated 11/1/2010.  Is it safe to delete ".fsr"?

-- 
Daniel

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Delay in folder access

[Stan Hoeppner]

On 7/23/2012 1:29 AM, Daniel L. Miller wrote:
> I have an XFS partition on simple 4-disk RAID-10 array.  The array is
> mounted at /raid.  I have a variety of subfolders - one of which is
> /raid/data - which in turn has a number of subfolders.
> 
> Doing a simple "ls /raid" has no problems.  Similarly, no issues with
> "ls /raid/data/stuff".  But ... executing "ls /raid/data" results in a
> delay, sometimes in minutes, before it returns.

How many files are in /raid/data, excluding subdirs?  If it's many tens
of thousands ls will be slow, but I'd not think multiple minutes worth
of slow.

Please post output of 'df /raid'

Likewise, 'xfs_db -c frag /dev/[device] -r'

device being that upon which the XFS resides.

> I don't see any errors in dmesg or syslog.  Does this indicate a pending
> crash?

Probably not.  It probably indicates a very full and fragmented filesystem.

> Also, I do have a ".fsr" folder under /raid.  All the folders listed,
> which appear empty, are dated 11/1/2010.  Is it safe to delete ".fsr"?

That may be related to xfs_fsr.  I've never seen such a file and I use
xfs_fsr regularly.  I'd wait for one of the devs to respond here before
deleting it.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Delay in folder access

[Dave Chinner]

On Mon, Jul 23, 2012 at 05:28:01AM -0500, Stan Hoeppner wrote:
> On 7/23/2012 1:29 AM, Daniel L. Miller wrote:
> > I have an XFS partition on simple 4-disk RAID-10 array.  The array is
> > mounted at /raid.  I have a variety of subfolders - one of which is
> > /raid/data - which in turn has a number of subfolders.
> > 
> > Doing a simple "ls /raid" has no problems.  Similarly, no issues with
> > "ls /raid/data/stuff".  But ... executing "ls /raid/data" results in a
> > delay, sometimes in minutes, before it returns.
> 
> How many files are in /raid/data, excluding subdirs?  If it's many tens
> of thousands ls will be slow, but I'd not think multiple minutes worth
> of slow.

depends. ls, by default, stats all the inodes as well to determine
what colour to make the output. If the inodes are randomly
distributed around the place, then ls can easily run at a seek per
file....

> Please post output of 'df /raid'
> 
> Likewise, 'xfs_db -c frag /dev/[device] -r'

Probably irrelevant.

What you really want is to determine inode location and directory
fragmentation, not the state of file fragmentatio. i.e. the inode
numbers of all the files and the xfs_bmap output of the directory
itself will give you an idea of the expected seek patterns on read...

And, of course, iostat -x -d -m 5 output will actually tell you what
the seek load is....

If it's a small directory, then there is something else happening,
and the output of 'echo w > /proc/sysrq-trigger' in dmesg woul dbe
useful. Yep, it's a FAQ and answer again:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

> > Also, I do have a ".fsr" folder under /raid.  All the folders listed,
> > which appear empty, are dated 11/1/2010.  Is it safe to delete ".fsr"?
> 
> That may be related to xfs_fsr. 

It is a directory left behind by xfs_fsr. it either crashed or was
killed without the proper signal handlers set up so left it behind.

> I've never seen such a file and I use
> xfs_fsr regularly.  I'd wait for one of the devs to respond here before
> deleting it.

>From xfsprogs:fsr/xfs_fsr.c:

/*
 * Initialize a directory for tmp file use.  This is used
 * by the full filesystem defragmentation when we're walking
 * the inodes and do not know the path for the individual
 * files.  Multiple directories are used to spread out the
 * tmp data around to different ag's (since file data is
 * usually allocated to the same ag as the directory and
 * directories allocated round robin from the same
 * parent directory).
 */
static void
tmp_init(char *mnt)
{
        int     i;
        static char     buf[SMBUFSZ];
        mode_t  mask;

        tmp_agi = 0;
        sprintf(buf, "%s/.fsr", mnt);
....

It's quite safe to delete it....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Delay in folder access

[Daniel L. Miller]

On 7/23/2012 3:28 AM, Stan Hoeppner wrote:
> On 7/23/2012 1:29 AM, Daniel L. Miller wrote:
>> I have an XFS partition on simple 4-disk RAID-10 array.  The array is
>> mounted at /raid.  I have a variety of subfolders - one of which is
>> /raid/data - which in turn has a number of subfolders.
>>
>> Doing a simple "ls /raid" has no problems.  Similarly, no issues with
>> "ls /raid/data/stuff".  But ... executing "ls /raid/data" results in a
>> delay, sometimes in minutes, before it returns.
> How many files are in /raid/data, excluding subdirs?  If it's many tens
> of thousands ls will be slow, but I'd not think multiple minutes worth
> of slow.
0 files, 7 subdirs, 1 symlink

>
> Please post output of 'df /raid'
df /raid
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/md0       312516096 215009428  97506668  69% /raid


> Likewise, 'xfs_db -c frag /dev/[device] -r'
xfs_db -c frag /dev/md0 -r
actual 282381, ideal 243258, fragmentation factor 13.85%

-- 
Daniel

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Delay in folder access

[Daniel L. Miller]

On 7/23/2012 10:44 AM, Daniel L. Miller wrote:
> On 7/23/2012 3:28 AM, Stan Hoeppner wrote:
>> On 7/23/2012 1:29 AM, Daniel L. Miller wrote:
>>> I have an XFS partition on simple 4-disk RAID-10 array.  The array is
>>> mounted at /raid.  I have a variety of subfolders - one of which is
>>> /raid/data - which in turn has a number of subfolders.
>>>
>>> Doing a simple "ls /raid" has no problems.  Similarly, no issues with
>>> "ls /raid/data/stuff".  But ... executing "ls /raid/data" results in a
>>> delay, sometimes in minutes, before it returns.
>> How many files are in /raid/data, excluding subdirs?  If it's many tens
>> of thousands ls will be slow, but I'd not think multiple minutes worth
>> of slow.
> 0 files, 7 subdirs, 1 symlink

I think I figured it out - and it has absolutely nothing to do with 
XFS.  That symlink?  It's to an autofs nfs mount - which doesn't exist.

Removing the symlink seems to have resolved the issue.
-- 
Daniel

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs