Hello, we have in the last few weeks always problems with the XFS file system. XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) After a certain time there is a kernel crash which looks as follows: September 23 10:17:57 hostname kernel: [1282081.010814] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:17:59 hostname kernel: [1282082.913437] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:01 hostname kernel: [1282084.908582] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:03 hostname kernel: [1282086.691791] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:05 hostname kernel: [1282088.691319] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:07 hostname kernel: [1282090.690182] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:09 hostname kernel: [1282092.673355] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:11 hostname kernel: [1282094.596504] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:13 hostname kernel: [1282096.595971] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:15 hostname kernel: [1282098.595301] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:17 hostname kernel: [1282100.594232] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:19 hostname kernel: [1282102.433262] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:21 hostname kernel: [1282104.412432] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:23 hostname kernel: [1282106.411598] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:25 hostname kernel: [1282108.310740] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:27 hostname kernel: [1282110.289892] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:29 hostname kernel: [1282112.289106] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:31 hostname kernel: [1282114.288247] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:33 hostname kernel: [1282116.215953] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:35 hostname kernel: [1282118.198488] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:37 hostname kernel: [1282120.193686] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:39 hostname kernel: [1282122.288716] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:41 hostname kernel: [1282124.215970] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:43 hostname kernel: [1282126.307031] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:45 hostname kernel: [1282128.306190] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:47 hostname kernel: [1282130.405310] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:49 hostname kernel: [1282132.076611] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:51 hostname kernel: [1282134.027790] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:53 hostname kernel: [1282136.122911] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:55 hostname kernel: [1282138.218032] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:57 hostname kernel: [1282140.145365] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:18:59 hostname kernel: [1282142.140528] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:19:01 hostname kernel: [1282144.091708] XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) September 23 10:19:01 hostname kernel: [1282144.223485] INFO: task kswapd0: 87 blocked for more than 120 seconds. September 23 10:19:01 hostname kernel: [1282144.224156] Not tainted 3.13.0-34-generic # 60-Ubuntu ~ precise1 September 23 10:19:01 hostname kernel: [1282144.224837] "echo 0> / proc / sys / kernel / hung_task_timeout_secs" disables this message. September 23 10:19:01 hostname kernel: [1282144.225597] kswapd0 D ffff88031662a020 0 87 2 0x00000000 September 23 10:19:01 hostname kernel: [1282144.225609] ffff88019500d8f8 0000000000000002 ffff880199d94440 ffff88019500dfd8 September 23 10:19:01 hostname kernel: [1282144.225621] 0000000000014440 0000000000014440 ffff880316cd2fe0 ffff880316f3afe0 September 23 10:19:01 hostname kernel: [1282144.225629] ffff88019500d970 ffff880314119128 ffff88031411912c 00000000ffffffff September 23 10:19:01 hostname kernel: [1282144.225638] Call Trace: September 23 10:19:01 hostname kernel: [1282144.225658] [] schedule + 0x29 / 0x70 September 23 10:19:01 hostname kernel: [1282144.225669] [] schedule_preempt_disabled + 0xe / 0x10 September 23 10:19:01 hostname kernel: [1282144.225680] [] __mutex_lock_slowpath + 0x114 / 0x1b0 September 23 10:19:01 hostname kernel: [1282144.225771] []? xfs_perag_get_tag + 0x41 / 0xf0 [xfs] September 23 10:19:01 hostname kernel: [1282144.225781] [] mutex_lock + 0x23 / 0x37 September 23 10:19:01 hostname kernel: [1282144.225832] [] xfs_reclaim_inodes_ag + 0x2e0 / 0x3a0 [xfs] September 23 10:19:01 hostname kernel: [1282144.225843] []? sched_clock + 0x9 / 0x10 September 23 10:19:01 hostname kernel: [1282144.225852] []? native_smp_send_reschedule + 0x4b / 0x60 September 23 10:19:01 hostname kernel: [1282144.225862] []? ttwu_queue + 0x8D / 0xD0 September 23 10:19:01 hostname kernel: [1282144.225871] []? try_to_wake_up + 0x190 / 0x210 September 23 10:19:01 hostname kernel: [1282144.225879] []? wake_up_process + 0x27 / 0x50 uname -a hostname 3.13.0-34-generic # 60-Ubuntu SMP Wed precise1 ~ August 13 15:55:33 UTC 2014 x86_64 x86_64 x86_64 GNU / Linux xfs_repair -V xfs_repair Version 3.1.7 7 CPUs -> cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 16 model : 4 model name : Quad-Core AMD Opteron(tm) Processor 2376 stepping : 2 microcode : 0x1000086 cpu MHz : 800.000 cache size : 512 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate npt lbrv svm_lock nrip_save bogomips : 4600.18 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate cat /proc/meminfo MemTotal: 12304884 kB MemFree: 151816 kB Buffers: 1076 kB Cached: 10328496 kB SwapCached: 0 kB Active: 2658232 kB Inactive: 7766540 kB Active(anon): 42644 kB Inactive(anon): 53636 kB Active(file): 2615588 kB Inactive(file): 7712904 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 4878268 kB SwapFree: 4878268 kB Dirty: 1220 kB Writeback: 0 kB AnonPages: 95728 kB Mapped: 32208 kB Shmem: 352 kB Slab: 1412904 kB SReclaimable: 1188188 kB SUnreclaim: 224716 kB KernelStack: 3512 kB PageTables: 28816 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 11030708 kB Committed_AS: 949792 kB VmallocTotal: 34359738367 kB VmallocUsed: 310312 kB VmallocChunk: 34353030140 kB HardwareCorrupted: 0 kB AnonHugePages: 12288 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 75520 kB DirectMap2M: 3069952 kB DirectMap1G: 9437184 kB cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=6108796k,nr_inodes=1527199,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=1230492k,mode=755 0 0 /dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f / xfs rw,relatime,attr2,inode64,noquota 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 /dev/sdc1 /extern/data0 xfs rw,relatime,attr2,inode64,noquota 0 0 rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 cat /proc/partitions major minor #blocks name 11 0 1048575 sr0 8 0 245175336 sda 8 1 97654784 sda1 8 2 4882432 sda2 8 16 245175336 sdb 8 17 97654784 sdb1 8 18 4882432 sdb2 9 2 97589120 md2 9 1 4878272 md1 8 32 27345461248 sdc 8 33 27345461214 sdc1 xfs_info /dev/md2 Metadaten =/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f isize=256 agcount=4, agsize=6099320 blks = sectsz=512 attr=2 Daten = bsize=4096 Blöcke=24397280, imaxpct=25 = sunit=0 swidth=0 blks Benennung =Version 2 bsize=4096 ascii-ci=0 Protokoll =Intern bsize=4096 Blöcke=11912, Version=2 = sectsz=512 sunit=0 blks, lazy-count=1 Echtzeit =keine extsz=4096 Blöcke=0, rtextents=0 xfs_info /dev/md2 Metadaten =/dev/disk/by-uuid/d6821e50-e82b-448d-bc03-9c286ef59c8f isize=256 agcount=4, agsize=6099320 blks = sectsz=512 attr=2 Daten = bsize=4096 Blöcke=24397280, imaxpct=25 = sunit=0 swidth=0 blks Benennung =Version 2 bsize=4096 ascii-ci=0 Protokoll =Intern bsize=4096 Blöcke=11912, Version=2 = sectsz=512 sunit=0 blks, lazy-count=1 Echtzeit =keine extsz=4096 Blöcke=0, rtextents=0 xfs_info /dev/sdc1 Metadaten =/dev/sdc1 isize=256 agcount=26, agsize=268435455 blks = sectsz=512 attr=2 Daten = bsize=4096 Blöcke=6836365303, imaxpct=5 = sunit=0 swidth=0 blks Benennung =Version 2 bsize=4096 ascii-ci=0 Protokoll =Intern bsize=4096 Blöcke=32768, Version=2 = sectsz=512 sunit=0 blks, lazy-count=0 Echtzeit =keine extsz=4096 Blöcke=0, rtextents=0 Can you help us? Best regards, -- Mit freundlichen Grüssen, Thomas Stagge -------------------------------------- Thomas Stagge, Dipl.Ing.(FH) System Administrator OPTIMARE Systems GmbH Am Luneort 15a 27572 Bremerhaven, Germany T: +49 471 48361-27 F: +49 471 48361-11 M: +49 172 8698575 thomas.stagge@optimare.de Internet: http://www.optimare.de CEO: Dipl.-Ing. Hans J. Stahl; Sitz der Gesellschaft: Bremerhaven; Registergericht: Amtsgericht Bremen HRB 28618 HB