* Re: dozens of xfsaild threads
2013-02-22 20:25 ` Ben Myers
@ 2013-02-22 23:27 ` Erik Knight
2013-02-24 11:13 ` Erik Knight
1 sibling, 0 replies; 4+ messages in thread
From: Erik Knight @ 2013-02-22 23:27 UTC (permalink / raw)
To: Ben Myers; +Cc: xfs
[-- Attachment #1.1: Type: text/plain, Size: 1460 bytes --]
Hi Ben,
Thanks for the help. The workload is a typical ETL processing. Building a
DataWarehouse. I'll get you the rest of that information as soon as I can
the tip about the metadata is already very interesting.
Thanks,
Erik
On Fri, Feb 22, 2013 at 9:25 PM, Ben Myers <bpm@sgi.com> wrote:
> Hi Erik,
>
> On Fri, Feb 22, 2013 at 02:04:17PM +0100, Erik Knight wrote:
> > We've recently noticed that our system is experiencing extreme
> performance
> > problems when running large workloads. The problem seems to come from
> > excessive System CPU time. Specifically dozens of xfsaild threads. We
> used
> > to have SSD drives but recently switched to HDD, so some of us are
> thinking
> > that there may be a configuration issue within XFS that is optimized for
> > SSD but performs terribly slow on HDD.
> >
> > Can anyone explain what these threads do, what would cause so many of
> them
> > to be running simultaneously or consume so much CPU?
>
> AIL stands for Active Item List. These guys sync metadata which has been
> logged to its final location on disk. You'll have one daemon per
> filesystem.
>
> If you have a very metadata intensive workload they could get a workout.
> It
> would help to know a bit more about your workload and configuration. Can
> you
> provide the relevant information listed here?
>
>
> http://www.xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>
> Regards,
> Ben
>
[-- Attachment #1.2: Type: text/html, Size: 2077 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: dozens of xfsaild threads
2013-02-22 20:25 ` Ben Myers
2013-02-22 23:27 ` Erik Knight
@ 2013-02-24 11:13 ` Erik Knight
1 sibling, 0 replies; 4+ messages in thread
From: Erik Knight @ 2013-02-24 11:13 UTC (permalink / raw)
To: Ben Myers; +Cc: xfs
[-- Attachment #1.1: Type: text/plain, Size: 5756 bytes --]
Hi Ben,
Thanks again for your help. This should be all the relevant information
that you requested:
The workload is classic ETL. The problem shows up when trying to refresh
the DWH.
*uname -a*
Linux 3.0.42-0.7-default #1 SMP Tue Oct 9 11:58:45 UTC 2012 (a8dc443)
x86_64 x86_64 x86_64 GNU/Linux
*xfs_repair -V*
xfs_repair version 3.1.8
Number of CPU: 8
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 2999.800
cache size : 6144 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dtherm
tpr_shadow vnmi flexpriority
bogomips : 6003.48
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
*cat /proc/meminfo*
MemTotal: 24738004 kB
MemFree: 903492 kB
Buffers: 364 kB
Cached: 22131464 kB
SwapCached: 28044 kB
Active: 15011808 kB
Inactive: 7694912 kB
Active(anon): 12375396 kB
Inactive(anon): 3024204 kB
Active(file): 2636412 kB
Inactive(file): 4670708 kB
Unevictable: 5160 kB
Mlocked: 5160 kB
SwapTotal: 18874364 kB
SwapFree: 13429788 kB
Dirty: 916 kB
Writeback: 0 kB
AnonPages: 559160 kB
Mapped: 13143856 kB
Shmem: 14821620 kB
Slab: 618944 kB
SReclaimable: 499808 kB
SUnreclaim: 119136 kB
KernelStack: 5512 kB
PageTables: 64124 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 31243364 kB
Committed_AS: 31705548 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 336820 kB
VmallocChunk: 34359375216 kB
HardwareCorrupted: 0 kB
AnonHugePages: 258048 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 129392 kB
DirectMap2M: 25034752 kB
*cat /proc/mounts*
rootfs / rootfs rw 0 0
udev /dev tmpfs rw,relatime,nr_inodes=0,mode=755 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/mapper/system-root / xfs rw,relatime,delaylog,noquota 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
/dev/cciss/c0d0p1 /boot ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
/dev/mapper/system-export /export xfs rw,relatime,delaylog,noquota 0 0
/dev/mapper/system-opt /opt xfs rw,relatime,delaylog,noquota 0 0
/dev/mapper/system-var /var xfs rw,relatime,delaylog,noquota 0 0
/dev/mapper/system-var /tmp xfs rw,relatime,delaylog,noquota 0 0
... etc
* cat /proc/partitions*
major minor #blocks name
104 0 143338560 cciss/c0d0
104 1 104391 cciss/c0d0p1
104 2 143227507 cciss/c0d0p2
253 0 10485760 dm-0
253 1 10485760 dm-1
253 2 8388608 dm-2
253 3 18874368 dm-3
253 4 5242880 dm-4
8 48 536870912 sdd
8 0 536870912 sda
8 64 536870912 sde
8 32 536870912 sdc
8 80 536870912 sdf
8 96 536870912 sdg
8 16 536870912 sdb
8 112 536870912 sdh
8 128 536870912 sdi
8 160 536870912 sdk
8 144 536870912 sdj
8 176 536870912 sdl
8 192 536870912 sdm
8 224 536870912 sdo
8 240 536870912 sdp
8 208 536870912 sdn
65 0 536870912 sdq
65 16 536870912 sdr
65 32 536870912 sds
65 48 536870912 sdt
65 64 536870912 sdu
65 80 536870912 sdv
65 96 536870912 sdw
65 112 536870912 sdx
65 128 536870912 sdy
65 144 536870912 sdz
65 160 536870912 sdaa
65 176 536870912 sdab
65 192 536870912 sdac
65 208 536870912 sdad
65 224 536870912 sdae
65 240 536870912 sdaf
253 5 536870912 dm-5
253 6 536870912 dm-6
etc....
On Fri, Feb 22, 2013 at 9:25 PM, Ben Myers <bpm@sgi.com> wrote:
> Hi Erik,
>
> On Fri, Feb 22, 2013 at 02:04:17PM +0100, Erik Knight wrote:
> > We've recently noticed that our system is experiencing extreme
> performance
> > problems when running large workloads. The problem seems to come from
> > excessive System CPU time. Specifically dozens of xfsaild threads. We
> used
> > to have SSD drives but recently switched to HDD, so some of us are
> thinking
> > that there may be a configuration issue within XFS that is optimized for
> > SSD but performs terribly slow on HDD.
> >
> > Can anyone explain what these threads do, what would cause so many of
> them
> > to be running simultaneously or consume so much CPU?
>
> AIL stands for Active Item List. These guys sync metadata which has been
> logged to its final location on disk. You'll have one daemon per
> filesystem.
>
> If you have a very metadata intensive workload they could get a workout.
> It
> would help to know a bit more about your workload and configuration. Can
> you
> provide the relevant information listed here?
>
>
> http://www.xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>
> Regards,
> Ben
>
[-- Attachment #1.2: Type: text/html, Size: 7932 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread