Slow filesystem.

Slow filesystem.

[Rogier Wolff]

Hi

My work (as in what brings in money to buy things from) file-system is
now "slow as hell". Last time this happened you guys told me that this
was because I had millions of hardlinks and way too many files.

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/md0                815M    4.1M    811M    1% /recover2

I have only 4.1M files. Only 1% inodes "in use". 


The filesystem was filling up lately, but i've managed to create some
free space.

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              6.3T  5.3T  720G  89% /recover2

 The question is: How do I get the filesystem to perform
normally again?


All mkdir calls are taking around 62 milliseconds: 

14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>

while on my workstation where i'm not supposed to be working
with client-data I get: 

14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>

So, things on my big datastorage drive are about 2000 times slower
than on my workstation.

	Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

Re: Slow filesystem.

[Eric Sandeen]

On 5/13/11 7:59 AM, Rogier Wolff wrote:
> 
> Hi
> 
> My work (as in what brings in money to buy things from) file-system is
> now "slow as hell". Last time this happened you guys told me that this
> was because I had millions of hardlinks and way too many files.
> 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/md0                815M    4.1M    811M    1% /recover2
> 
> I have only 4.1M files. Only 1% inodes "in use". 
> 
> 
> The filesystem was filling up lately, but i've managed to create some
> free space.
> 
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md0              6.3T  5.3T  720G  89% /recover2
> 
>  The question is: How do I get the filesystem to perform
> normally again?
> 
> 
> All mkdir calls are taking around 62 milliseconds: 
> 
> 14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>

I don't know what the workload is, and "slow" isn't very well characterized here, but from this can I assume that mkdir is the bottleneck?

Did you ever try that patch I sent you for this issue in March?

Using oprofile to see where the fs is spending its time might also be useful.

-Eric

> while on my workstation where i'm not supposed to be working
> with client-data I get: 
> 
> 14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
> 
> So, things on my big datastorage drive are about 2000 times slower
> than on my workstation.
> 
> 	Roger. 
> 

Re: Slow filesystem.

[Andreas Dilger]

On May 13, 2011, at 06:59, Rogier Wolff wrote:
> My work (as in what brings in money to buy things from) file-system is
> now "slow as hell". Last time this happened you guys told me that this
> was because I had millions of hardlinks and way too many files.
> 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/md0                815M    4.1M    811M    1% /recover2
> 
> I have only 4.1M files. Only 1% inodes "in use". 
> 
> 
> The filesystem was filling up lately, but i've managed to create some
> free space.
> 
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md0              6.3T  5.3T  720G  89% /recover2

I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks.  That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.

> The question is: How do I get the filesystem to perform
> normally again?

It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.

> All mkdir calls are taking around 62 milliseconds: 
> 
> 14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312>
> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658>
> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>
> 
> while on my workstation where i'm not supposed to be working
> with client-data I get: 
> 
> 14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033>
> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031>
> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
> 
> So, things on my big datastorage drive are about 2000 times slower
> than on my workstation.
> 
> 	Roger. 
> 
> -- 
> ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
> **    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
> *-- BitWizard writes Linux device drivers for any device you may have! --*
> Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
> Does it sit on the couch all day? Is it unemployed? Please be specific! 
> Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas

Re: Slow filesystem.

[Rogier Wolff]

On Fri, May 13, 2011 at 11:21:52PM -0600, Andreas Dilger wrote:
> On May 13, 2011, at 06:59, Rogier Wolff wrote:
> > My work (as in what brings in money to buy things from) file-system is
> > now "slow as hell". Last time this happened you guys told me that this
> > was because I had millions of hardlinks and way too many files.
> > 
> > Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> > /dev/md0                815M    4.1M    811M    1% /recover2
> > 
> > I have only 4.1M files. Only 1% inodes "in use". 
> > 
> > 
> > The filesystem was filling up lately, but i've managed to create some
> > free space.
> > 
> > Filesystem            Size  Used Avail Use% Mounted on
> > /dev/md0              6.3T  5.3T  720G  89% /recover2

> I suspect that the free inodes in your filesystem are very
> fragmented, due to the filesystem filling up and then deleting a lot
> of files that create "holes" of free inodes within the inode table
> blocks.  That means that possibly finding free inodes is a bunch of
> work, and then possibly only one or two inodes are allocated from
> each block.

How can the free inodes be fragmented?!? 99% of the available inodes
ARE free.

> > The question is: How do I get the filesystem to perform
> > normally again?
 
> It is probably worthwhile to do a blktrace to see what the IO is
> looking like, and as Eric suggested the oprofile information may be
> interesting if a lot of CPU is being used.

A lot of CPU time is being used, yes. The 62 milliseconds are almost
completely CPU time.

I was running a big "backup" (image of a 500Gb disk) onto the
partition. Now that this has finished I'll try booting into a recent
kernel. Next I'll try finding and applying the patch.... 

	Roger.

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ