Slow file stat/deletion

Slow file stat/deletion

[Gionatan Danti]

Hi all,
I am using a XFS filesystem as a target for rsnapshot hardlink-based 
backups.

Being hardlink-based, our backups generally are quite fast. However, I 
noticed that for some directories having many small files the longer 
part of the backup process is to remove the old (out-of-retention) 
subdirs that must be purged to make room for the new backup iteration.

Further analysis show that the slower part of the 'rm' process is the 
reading of the affected inodes/dentries. An example: to remove a subdir 
with ~700000 files and directories, the system needs about 30 minutes. 
At the same time, issuing a simple "find <dir> / | wc -l" (after having 
dropped the caches) need ~24 minutes. In other words, actual reads need 
4x the real delete time.

So, my question is: there is anything I can do to speedup the 
read/stat/deletion?

Here is my system config:
CPU: AMD Opteron(tm) Processor 4334
RAM: 16 GB
HDD: 12x 2TB WD RE in a RAID6 array (64k stripe unit), attached to a 
PERC H700 controller with 512MB BBU writeback cache
OS:  CentOS 7.2 x86_64 with 3.10.0-327.18.2.el7.x86_64 kernel

Relevant LVM setup:
LV           VG         Attr       LSize  Pool         Origin Data% 
Meta%  Move Log Cpy%Sync Convert Chunk
000-ThinPool vg_storage twi-aotz-- 10,85t                     86,71 
38,53                            8,00m
Storage      vg_storage Vwi-aotz-- 10,80t 000-ThinPool        87,12 
                                 0

XFS filesystem info:
meta-data=/dev/mapper/vg_storage-Storage isize=512    agcount=32, 
agsize=90596992 blks
          =                       sectsz=512   attr=2, projid32bit=1
          =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=2899103744, imaxpct=5
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=521728, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Some consideration:
1) I am using a single big thin volume because back in the time ( >2 
years ago) I was not sure about XFS and, having no shrinking capability, 
I relayed on thin volume unmap should the filesystem choice change. 
However, thin pool's chunk size is quite big (8 MB) so it should not 
pose acute fragmentation problem;

2) due to being layered over a thinly provided volume, the filesystem 
was created with "noalign" option. I run some in-the-lab test on a spare 
machine and I (still) find that this option seems to *lower* the time 
needed to stat/delete files when XFS is on top of a thin volume, so I do 
not think this is a problem. I'm right?

3) the filesystem is over 2 years old and has a very big number of files 
on it (inode count is 12588595, but each inode has multiple hardlinked 
files). Is this slow delete performance a side effect of "aging" ?

4) I have not changed the default read-ahead value (256 KB). I know this 
is quite small compared to available disk resources but, before messing 
with low-level block device tuning, I would really like to know your 
opinion on my case.

Thank you all.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Slow file stat/deletion

[Dave Chinner]

On Fri, Nov 25, 2016 at 11:40:40AM +0100, Gionatan Danti wrote:
> Hi all,
> I am using a XFS filesystem as a target for rsnapshot hardlink-based
> backups.
> 
> Being hardlink-based, our backups generally are quite fast. However,
> I noticed that for some directories having many small files the
> longer part of the backup process is to remove the old
> (out-of-retention) subdirs that must be purged to make room for the
> new backup iteration.

Ah, hard link farms. aka "How to fragment the AGI btrees for fun and
profit."

> Further analysis show that the slower part of the 'rm' process is
> the reading of the affected inodes/dentries. An example: to remove a
> subdir with ~700000 files and directories, the system needs about 30
> minutes. At the same time, issuing a simple "find <dir> / | wc -l"
> (after having dropped the caches) need ~24 minutes. In other words,
> actual reads need 4x the real delete time.
> 
> So, my question is: there is anything I can do to speedup the
> read/stat/deletion?

Not now. Speed is a factor of the inode layout and seek times. Find
relies on sequential directory access which is sped up on XFS by
internal btree readahead and it doesn't require reading the extent
list. rm processes inodes one at a time and requires reading of the
extent list so per-inode there is more IO, a lot more CPU time spent
and more per-op latency, so it's no surprise it's much slower than
find.

> 
> Here is my system config:
> CPU: AMD Opteron(tm) Processor 4334
> RAM: 16 GB
> HDD: 12x 2TB WD RE in a RAID6 array (64k stripe unit), attached to a
> PERC H700 controller with 512MB BBU writeback cache
> OS:  CentOS 7.2 x86_64 with 3.10.0-327.18.2.el7.x86_64 kernel
> 
> Relevant LVM setup:
> LV           VG         Attr       LSize  Pool         Origin Data%
> Meta%  Move Log Cpy%Sync Convert Chunk
> 000-ThinPool vg_storage twi-aotz-- 10,85t                     86,71
> 38,53                            8,00m
> Storage      vg_storage Vwi-aotz-- 10,80t 000-ThinPool        87,12
> 0
> 
> XFS filesystem info:
> meta-data=/dev/mapper/vg_storage-Storage isize=512    agcount=32,
> agsize=90596992 blks
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=0        finobt=0

finobt=0.

finobt was added primarily to solve inode allocation age-related
degradation for hard link farm style workloads. It will have
significant impact on unlink as well, because initial inode
allocation patterns will be better...

> Some consideration:
> 1) I am using a single big thin volume because back in the time ( >2
> years ago) I was not sure about XFS and, having no shrinking
> capability, I relayed on thin volume unmap should the filesystem
> choice change. However, thin pool's chunk size is quite big (8 MB)
> so it should not pose acute fragmentation problem;

Nope, but it means that what should be sequential IO is probably
going to be random. i.e. instead of directory/inode/extent reading
IO having minimum track-track seek latency because they are all
nearby (1-2ms), they'll be average seeks (6-7ms) because locality is no
longer as the filesystem has optimised for.

> 
> 2) due to being layered over a thinly provided volume, the
> filesystem was created with "noalign" option.

noalign affects data placement only, and only for filesystems that
have a stripe unit/width set, which yours doesn't:

>			sunit=0      swidth=0 blks

> I run some in-the-lab
> test on a spare machine and I (still) find that this option seems to
> *lower* the time needed to stat/delete files when XFS is on top of a
> thin volume, so I do not think this is a problem. I'm right?

It's not a problem because it doesn't do anything with your fs
config.

> 3) the filesystem is over 2 years old and has a very big number of
> files on it (inode count is 12588595, but each inode has multiple
> hardlinked files). Is this slow delete performance a side effect of
> "aging" ?

Yes. Made worse by being on a thinp volume.


> 4) I have not changed the default read-ahead value (256 KB). I know
> this is quite small compared to available disk resources but, before
> messing with low-level block device tuning, I would really like to
> know your opinion on my case.

Only used for data readahead. Will make no difference to
directory/stat/unlink performance.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Slow file stat/deletion

[Gionatan Danti]

On 27/11/2016 23:14, Dave Chinner wrote:
>
> Ah, hard link farms. aka "How to fragment the AGI btrees for fun and
> profit."
>

Interesting... there is anything I can read about AGI fragmentation?

>
> Not now. Speed is a factor of the inode layout and seek times. Find
> relies on sequential directory access which is sped up on XFS by
> internal btree readahead and it doesn't require reading the extent
> list. rm processes inodes one at a time and requires reading of the
> extent list so per-inode there is more IO, a lot more CPU time spent
> and more per-op latency, so it's no surprise it's much slower than
> find.
>

To tell the truth, "find" and "rm" show quite similar results: ~24 
minutes for the former, ~30 minutes for the latter. I perfectly 
understand that "rm" is going to be slower than find; my point is that 
*even* "find" seems quite slow...

>
> finobt=0.
>
> finobt was added primarily to solve inode allocation age-related
> degradation for hard link farm style workloads. It will have
> significant impact on unlink as well, because initial inode
> allocation patterns will be better...
>

This is a very interesting information; thank you.

>
> Nope, but it means that what should be sequential IO is probably
> going to be random. i.e. instead of directory/inode/extent reading
> IO having minimum track-track seek latency because they are all
> nearby (1-2ms), they'll be average seeks (6-7ms) because locality is no
> longer as the filesystem has optimised for.
>

Should not thinp overhead be minimized by the big (8 MB) chunk size? Are 
inode allocation so much scattered around LBAs? Maybe the slowdown can 
be increased by bad journal placement (I imagine it is near the start of 
the disk, while current read/write activity surely happen near the end)?

>
> noalign affects data placement only, and only for filesystems that
> have a stripe unit/width set, which yours doesn't:
>
>> 			sunit=0      swidth=0 blks

Isn't that the proper results of "noalign"? By opting for "noalign" I am 
telling mkfs to discard any stripe information, right?

>
> Yes. Made worse by being on a thinp volume.
>

I can't do anything for that?

>
> Only used for data readahead. Will make no difference to
> directory/stat/unlink performance.

Thank you again for valuable information.

>
> Cheers,
>
> Dave.
>

Thanks Dave.


-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Slow file stat/deletion

[Dave Chinner]

On Mon, Nov 28, 2016 at 10:51:42AM +0100, Gionatan Danti wrote:
> 
> 
> On 27/11/2016 23:14, Dave Chinner wrote:
> >
> >Ah, hard link farms. aka "How to fragment the AGI btrees for fun and
> >profit."
> >
> 
> Interesting... there is anything I can read about AGI fragmentation?

Read up on finobt and the bug reports on the list about how inode
allocation slows to a crawl....

> >Nope, but it means that what should be sequential IO is probably
> >going to be random. i.e. instead of directory/inode/extent reading
> >IO having minimum track-track seek latency because they are all
> >nearby (1-2ms), they'll be average seeks (6-7ms) because locality is no
> >longer as the filesystem has optimised for.
> >
> 
> Should not thinp overhead be minimized by the big (8 MB) chunk size?

Minimised - maybe. Removed - no.

> Are inode allocation so much scattered around LBAs?

Yes. XFS distributes inodes across the entire device LBA.

> Maybe the
> slowdown can be increased by bad journal placement (I imagine it is
> near the start of the disk, while current read/write activity surely
> happen near the end)?

Contributing factor, yes. You just have to live with that thinp
behaviour.

> >noalign affects data placement only, and only for filesystems that
> >have a stripe unit/width set, which yours doesn't:
> >
> >>			sunit=0      swidth=0 blks
> 
> Isn't that the proper results of "noalign"?

No. "noalign" is a mount option - the sunit/swidth are geometry
values stored in the superblock. noalign will override the
superblock values, but it does not make them go away.

> By opting for "noalign"
> I am telling mkfs to discard any stripe information, right?

No. You are telling it to ignore stripe alignment for file data
allocation purposes.

> >Yes. Made worse by being on a thinp volume.
> 
> I can't do anything for that?

Nope. There's always going to be a penalty for subverting the
filesystem's physical layout optimisations on storage subsystems
that require physical layout optimisation for performance.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Slow file stat/deletion

[Gionatan Danti]

On 28/11/2016 22:53, Dave Chinner wrote:
>
> Nope. There's always going to be a penalty for subverting the
> filesystem's physical layout optimisations on storage subsystems
> that require physical layout optimisation for performance.
>
> Cheers,
>
> Dave.
>

Ok, very clear.

Thank you for taking the time to explain, Dave.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Slow file stat/deletion

[Gionatan Danti]

Hi all,
this is a note which can be useful to someone searching for the same 
thing: using "ftype=1" at mkfs time (which is, by the way, the default 
for CRC-enabled filesystem) greatly speed-up stat/find operations.

On a test case with 1.000.000 files and 1000 total directories, "find 
/mnt/xfs" time has fallen from ~70 to ~20 seconds.

As a side question: there is no method/possibility to enable ftype=1 to 
an already existing filesystem, right?
Thanks.

Il 29-11-2016 08:53 Gionatan Danti ha scritto:
> On 28/11/2016 22:53, Dave Chinner wrote:
>> 
>> Nope. There's always going to be a penalty for subverting the
>> filesystem's physical layout optimisations on storage subsystems
>> that require physical layout optimisation for performance.
>> 
>> Cheers,
>> 
>> Dave.
>> 
> 
> Ok, very clear.
> 
> Thank you for taking the time to explain, Dave.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: Slow file stat/deletion

[Eric Sandeen]

On 4/28/17 3:14 PM, Gionatan Danti wrote:
> Hi all,
> this is a note which can be useful to someone searching for the same thing: using "ftype=1" at mkfs time (which is, by the way, the default for CRC-enabled filesystem) greatly speed-up stat/find operations.
> 
> On a test case with 1.000.000 files and 1000 total directories, "find /mnt/xfs" time has fallen from ~70 to ~20 seconds.
> 
> As a side question: there is no method/possibility to enable ftype=1 to an already existing filesystem, right?
> Thanks.

That is correct.

-Eric