Re: extremely slow file creation/deletion after xfs ran full

[Carsten Aulbert <Carsten.Aulbert@aei.mpg.de>]

[-- Attachment #1: Type: text/plain, Size: 2350 bytes --]

Hi again

(sorry that I reply to my own email, rather than Brian's, but I've only
just subscribed to the list), I'll try to address Brian's email in here
with fake quotation. Sorry for breaking the threading :(

Brian Foster wrote:
> Based on the size and consumption of the fs, first thing that comes
> to mind is perhaps fragmented use of inode records. E.g., inode
> records are spread all over the storage with handfulls of inodes
> free here and there, which means individual inode allocation can
> take a hit searching for the record with free inodes. I don't think
> that explains rm performance though.

> It might be interesting to grab a couple perf traces of the touch
> and rm commands and see what cpu usage looks like. E.g., 'perf
> record -g touch <file>,' 'perf report -g,' etc.

I've attached both perf outputs and after reviewing them briefly I think
slowness is caused by different means, i.e. only the touch one is in
xfs' territory.

> 	30265117 xfs: Fix rounding in xfs_alloc_fix_len()
>
> That originally went into 3.16 and I don't see it in the 3.14 stable
> branch. Did xfs_repair actually report anything wrong?

Nope, only displayed all the stages, but nothing was fixed.

> It seems like you have sufficiently large and available free
> space. That said, it's fairly common for filesytems to naturally
> drop in performance as free space becomes more limited. E.g., I
> think it's common practice to avoid regular usage while over 80%
> used if performance is a major concern. Also, I doubt xfs_fsr will
> do much to affect inode allocation performance, but I could be
> wrong.

Yes, we should have monitored that mount point rather than /tmp which we
did when bad things happened(TM). Given that we have a high
fragmentation of directories, would xfs_fsr help here at all?

Regarding v5, currently we are copying data off that disk and will
create it anew with -m crc=1,finobt=1 on a recent 3.18 kernel. Apart
from that I don't know much we can further do to safe-guard us against
this happening again (well kepp it below 80% all the time as well).

Thanks a lot for the remarks!

cheers

Carsten

-- 
Dr. Carsten Aulbert - Max Planck Institute for Gravitational Physics
Callinstrasse 38, 30167 Hannover, Germany
phone/fax: +49 511 762-17185 / -17193
https://wiki.atlas.aei.uni-hannover.de/foswiki/bin/view/ATLAS/WebHome

[-- Attachment #2: rm.perf.data.gz --]
[-- Type: application/gzip, Size: 2822 bytes --]

[-- Attachment #3: touch.perf.data.gz --]
[-- Type: application/gzip, Size: 14833 bytes --]

[-- Attachment #4: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs