Issues and new to the group

Ronnie Tartar
2013-09-26T11:47:25Z

Hi,

I have a 600GB xfs file system mounted that suddenly started running slow on
writes. It takes about 2.5 to 3.5 seconds to write a single file. Some
folders (with less number of files) work well. But it will copy fast, then
slow for long periods of time. This is a virtualized CentOS 5.9 64 bit box
on Citrix Xenserver 5.6SP2. Doesn't seem to be a load i/o issue as most of
the load is system%. My fragmentation is less than 1 %. Any help would
be greatly appreciated. I was looking to see if there was a better way to
mount this partition or allocate more memory, whatever it takes. The
folders are image folders that have anywhere between 5 to 10 million images
in each folder.

Thanks

Fstab mount is:
/dev/xvdb1 /images xfs
defaults,nodiratime,nosuid,nodev,allocsize=64m 1 1

Slabtop is:
Active / Total Objects (% used) : 2705947 / 2872142 (94.2%)
Active / Total Slabs (% used) : 290008 / 290008 (100.0%)
Active / Total Caches (% used) : 111 / 165 (67.3%)
Active / Total Size (% used) : 1048796.52K / 1083850.55K (96.8%)
Minimum / Average / Maximum Object : 0.02K / 0.38K / 128.00K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME

843080 836242 99% 0.44K 105385 8 421540K xfs_inode
839608 836242 99% 0.56K 119944 7 479776K xfs_vnode
461610 432085 93% 0.21K 25645 18 102580K dentry_cache
306200 306200 100% 0.09K 7655 40 30620K buffer_head
222570 121732 54% 0.12K 7419 30 29676K size-128
117439 97108 82% 0.52K 16777 7 67108K radix_tree_node
30820 30814 99% 0.19K 1541 20 6164K xfs_ili
13270 13266 99% 0.74K 2654 5 10616K ext3_inode_cache
9390 9390 100% 0.25K 626 15 2504K size-256
7682 7562 98% 0.16K 334 23 1336K vm_area_struct
3068 1711 55% 0.06K 52 59 208K size-64
2816 2786 98% 0.09K 64 44 256K sysfs_dir_cache
2055 1275 62% 0.25K 137 15 548K filp
1440 1289 89% 0.02K 10 144 40K anon_vma
1120 999 89% 0.03K 10 112 40K size-32
768 534 69% 0.08K 16 48 64K
selinux_inode_security
756 698 92% 0.55K 108 7 432K inode_cache
576 554 96% 0.58K 96 6 384K proc_inode_cache
476 455 95% 1.00K 119 4 476K size-1024
404 403 99% 2.00K 202 2 808K size-2048
404 404 100% 4.00K 404 1 1616K size-4096
360 350 97% 0.12K 12 30 48K bio
320 284 88% 0.50K 40 8 160K size-512