* 10GB memorys occupied by XFS
@ 2014-04-04 7:26 daiguochao
2014-04-04 20:24 ` Stan Hoeppner
2014-04-11 2:40 ` daiguochao
0 siblings, 2 replies; 7+ messages in thread
From: daiguochao @ 2014-04-04 7:26 UTC (permalink / raw)
To: xfs
Hello folks,
I used xfs file system in kernel-2.6.32-220.13.1.el6.x86_64 for store
pictures. About 100 days system memorys is lost and some nginx process is
killed by oom-killer.So,I looked /proc/meminfo and find memorys is
lost.Finally, I try to umount xfs system and 10GB memorys is coming back. l
look xfs bugzilla no such BUG.I have no idea for it.
Cheers,
Guochao.
some memorys info:
0> free -m
total used free shared buffers cached
Mem: 11887 11668 219 0 0 2
-/+ buffers/cache: 11665 222
Swap: 0 0 0
130> cat /proc/meminfo
MemTotal: 12173268 kB
MemFree: 223044 kB
Buffers: 244 kB
Cached: 4540 kB
SwapCached: 0 kB
Active: 1700 kB
Inactive: 5312 kB
Active(anon): 1616 kB
Inactive(anon): 1128 kB
Active(file): 84 kB
Inactive(file): 4184 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2556 kB
Mapped: 1088 kB
Shmem: 196 kB
Slab: 509708 kB
SReclaimable: 7596 kB
SUnreclaim: 502112 kB
KernelStack: 1096 kB
PageTables: 748 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6086632 kB
Committed_AS: 9440 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 303488 kB
VmallocChunk: 34359426132 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6152 kB
DirectMap2M: 2070528 kB
DirectMap1G: 10485760 kB
--
View this message in context: http://xfs.9218.n7.nabble.com/10GB-memorys-occupied-by-XFS-tp35015.html
Sent from the Xfs - General mailing list archive at Nabble.com.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 10GB memorys occupied by XFS
2014-04-04 7:26 10GB memorys occupied by XFS daiguochao
@ 2014-04-04 20:24 ` Stan Hoeppner
[not found] ` <76016fc7.13c84.14546bad411.Coremail.dx-wl@163.com>
2014-04-11 2:40 ` daiguochao
1 sibling, 1 reply; 7+ messages in thread
From: Stan Hoeppner @ 2014-04-04 20:24 UTC (permalink / raw)
To: daiguochao, xfs
On 4/4/2014 2:26 AM, daiguochao wrote:
> Hello folks,
Hello,
Note that your problems are not XFS specific, but can occur with any
Linux filesystem.
> I used xfs file system in kernel-2.6.32-220.13.1.el6.x86_64 for store
> pictures. About 100 days system memorys is lost and some nginx process is
> killed by oom-killer.So,I looked /proc/meminfo and find memorys is
> lost.Finally, I try to umount xfs system and 10GB memorys is coming back. l
> look xfs bugzilla no such BUG.I have no idea for it.
>
> Cheers,
>
> Guochao.
>
> some memorys info:
>
> 0> free -m
> total used free shared buffers cached
> Mem: 11887 11668 219 0 0 2
> -/+ buffers/cache: 11665 222
> Swap: 0 0 0
First problem: no swap
Second problem: cache is not being reclaimed
Read vfs_cache_pressure at:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
You've likely set this value to zero. Changing it to 200 should prompt
the kernel to reclaim dentries and inodes aggressively, preventing the
oom-killer from kicking in.
Cheers,
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* 答复: Re: 10GB memorys occupied by XFS
[not found] ` <76016fc7.13c84.14546bad411.Coremail.dx-wl@163.com>
@ 2014-04-10 1:41 ` 戴国超
2014-04-11 5:09 ` Stan Hoeppner
1 sibling, 0 replies; 7+ messages in thread
From: 戴国超 @ 2014-04-10 1:41 UTC (permalink / raw)
To: stan@hardwarefreak.com; +Cc: xfs@oss.sgi.com
Dear Stan,
Thank you for your kind assistance.
In accordance with your suggestion, we executed "echo 3 > /proc/sysm/drop_caches" for trying to release vfs dentries and inodes. Really, our lost memory came back. But we learned that the memory of vfs dentries and inodes is distributed from slab. Please check our system "Slab: 509708 kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take up 450MB among. And /proc/meminfo indicated that our system memory is anomalous, there is about 10GB out of the statistics. We want to know how the system could observe the usage amount of vfs dentries and iodes through the system interface. If the memory usage of system is not reflected in /proc/meminfo as we can not find the statistics, and we thought it as a bug of xfs.
My vm.vfs_cache_pressure of linux system is 100. We think that the system will proactively take the memory back when the memory is not enough, rather than oom-killer kills our work process. Our datas of /proc/meminfo occurred during the system problem as below:
130> cat /proc/meminfo
MemTotal: 12173268 kB
MemFree: 223044 kB
Buffers: 244 kB
Cached: 4540 kB
SwapCached: 0 kB
Active: 1700 kB
Inactive: 5312 kB
Active(anon): 1616 kB
Inactive(anon): 1128 kB
Active(file): 84 kB
Inactive(file): 4184 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2556 kB
Mapped: 1088 kB
Shmem: 196 kB
Slab: 509708 kB
SReclaimable: 7596 kB
SUnreclaim: 502112 kB
KernelStack: 1096 kB
PageTables: 748 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6086632 kB
Committed_AS: 9440 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 303488 kB
VmallocChunk: 34359426132 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6152 kB
DirectMap2M: 2070528 kB
DirectMap1G: 10485760 kB
I look forward to hearing from you and thank you very much for your kind assistance.
Best Regards,
Guochao
At 2014-04-05 04:24:12,"Stan Hoeppner" <stan@hardwarefreak.com> wrote:
>On 4/4/2014 2:26 AM, daiguochao wrote:
>> Hello folks,
>
>Hello,
>
>Note that your problems are not XFS specific, but can occur with any
>Linux filesystem.
>
>> I used xfs file system in kernel-2.6.32-220.13.1.el6.x86_64 for store
>> pictures. About 100 days system memorys is lost and some nginx
>>process is
>> killed by oom-killer.So,I looked /proc/meminfo and find memorys is
>> lost.Finally, I try to umount xfs system and 10GB memorys is coming
>>back. l
>> look xfs bugzilla no such BUG.I have no idea for it.
>>
>> Cheers,
>>
>> Guochao.
>>
>> some memorys info:
>>
>> 0> free -m
>> total used free shared buffers
>>cached
>> Mem: 11887 11668 219 0 0
>>2
>> -/+ buffers/cache: 11665 222
>> Swap: 0 0 0
>
>
>First problem: no swap
>Second problem: cache is not being reclaimed
>
>Read vfs_cache_pressure at:
>https://www.kernel.org/doc/Documentation/sysctl/vm.txt
>
>You've likely set this value to zero. Changing it to 200 should prompt
>the kernel to reclaim dentries and inodes aggressively, preventing the
>oom-killer from kicking in.
>
>Cheers,
>
>Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 10GB memorys occupied by XFS
2014-04-04 7:26 10GB memorys occupied by XFS daiguochao
2014-04-04 20:24 ` Stan Hoeppner
@ 2014-04-11 2:40 ` daiguochao
2014-04-11 4:26 ` Dave Chinner
2014-04-11 21:35 ` Stan Hoeppner
1 sibling, 2 replies; 7+ messages in thread
From: daiguochao @ 2014-04-11 2:40 UTC (permalink / raw)
To: xfs
Dear Stan, I can't send email to you.So I leave a message here.I hope not to
bother you.
Thank you for your kind assistance.
In accordance with your suggestion, we executed "echo 3 >
/proc/sysm/drop_caches" for trying to release vfs dentries and inodes.
Really,
our lost memory came back. But we learned that the memory of vfs dentries
and inodes is distributed from slab. Please check our system "Slab: 509708
kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take
up 450MB among. And /proc/meminfo indicated that our system memory is
anomalous, there is about 10GB out of the statistics. We want to know how
the system could observe the usage amount of vfs dentries and iodes through
the system interface. If the memory usage of system is not reflected in
/proc/meminfo as we can not find the statistics, and we thought it as a bug
of xfs.
My vm.vfs_cache_pressure of linux system is 100. We think that the system
will proactively take the memory back when the memory is not enough, rather
than oom-killer kills our work process. Our datas of /proc/meminfo occurred
during the system problem as below:
130> cat /proc/meminfo
MemTotal: 12173268 kB
MemFree: 223044 kB
Buffers: 244 kB
Cached: 4540 kB
SwapCached: 0 kB
Active: 1700 kB
Inactive: 5312 kB
Active(anon): 1616 kB
Inactive(anon): 1128 kB
Active(file): 84 kB
Inactive(file): 4184 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2556 kB
Mapped: 1088 kB
Shmem: 196 kB
Slab: 509708 kB
SReclaimable: 7596 kB
SUnreclaim: 502112 kB
KernelStack: 1096 kB
PageTables: 748 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 6086632 kB
Committed_AS: 9440 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 303488 kB
VmallocChunk: 34359426132 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6152 kB
DirectMap2M: 2070528 kB
DirectMap1G: 10485760 kB
Best Regards,
Guochao
--
View this message in context: http://xfs.9218.n7.nabble.com/10GB-memorys-occupied-by-XFS-tp35015p35016.html
Sent from the Xfs - General mailing list archive at Nabble.com.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 10GB memorys occupied by XFS
2014-04-11 2:40 ` daiguochao
@ 2014-04-11 4:26 ` Dave Chinner
2014-04-11 21:35 ` Stan Hoeppner
1 sibling, 0 replies; 7+ messages in thread
From: Dave Chinner @ 2014-04-11 4:26 UTC (permalink / raw)
To: daiguochao; +Cc: xfs
On Thu, Apr 10, 2014 at 07:40:44PM -0700, daiguochao wrote:
> Dear Stan, I can't send email to you.So I leave a message here.I hope not to
> bother you.
> Thank you for your kind assistance.
>
> In accordance with your suggestion, we executed "echo 3 >
> /proc/sysm/drop_caches" for trying to release vfs dentries and inodes.
> Really,
> our lost memory came back. But we learned that the memory of vfs dentries
> and inodes is distributed from slab. Please check our system "Slab: 509708
> kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take
> up 450MB among.
That's where your memory is - in metadata buffers. The xfs_buf slab
entries are just the handles - the metadata pages in the buffers
usually take much more space and it's not accounted to the slab
cache nor the page cache.
Can you post the output of /proc/slabinfo, and what is the output of
xfs_info on the filesystem in question? Also, a description of your
workload that is resulting in large amounts of cached metadata
buffers but no inodes or dentries would be helpful.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 10GB memorys occupied by XFS
[not found] ` <76016fc7.13c84.14546bad411.Coremail.dx-wl@163.com>
2014-04-10 1:41 ` 答复: " 戴国超
@ 2014-04-11 5:09 ` Stan Hoeppner
1 sibling, 0 replies; 7+ messages in thread
From: Stan Hoeppner @ 2014-04-11 5:09 UTC (permalink / raw)
To: 戴国超; +Cc: xfs
On 4/9/2014 8:43 AM, 戴国超 wrote:
> Dear Stan,
> Thank you for your kind assistance.
>
> In accordance with your suggestion, we executed "echo 3 > /proc/sysm/drop_caches" for trying to release vfs dentries and inodes. Really,
> our lost memory came back. But we learned that the memory of vfs dentries and inodes is distributed from slab. Please check our system "Slab: 509708 kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take up 450MB among.
To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
echo 3 > /proc/sys/vm/drop_caches
> And /proc/meminfo indicated that our system memory is anomalous, there is about 10GB out of the statistics. We want to know how the system could observe the usage amount of vfs dentries and iodes through the system interface. If the memory usage of system is not reflected in /proc/meminfo as we can not find the statistics, and we thought it as a bug of xfs.
It seems much of this 10 GB of memory is being consumed by pagecache,
not dentries and inodes. So the question is:
Why is pagecache not being reclaimed without manual intervention?
> My vm.vfs_cache_pressure of linux system is 100. We think that the system will proactively take the memory back when the memory is not enough, rather than oom-killer kills our work process. Our datas of /proc/meminfo occurred during the system problem as below:
Except most of your slab is reported as not reclaimable, see below.
> 130> cat /proc/meminfo
> MemTotal: 12173268 kB
> MemFree: 223044 kB
> Buffers: 244 kB
> Cached: 4540 kB <------
4.5 MB is reported use by the pagecache. But some 10 GB is being
consumed by the page cache and not reported here.
> SwapCached: 0 kB
> Active: 1700 kB
> Inactive: 5312 kB
> Active(anon): 1616 kB
> Inactive(anon): 1128 kB
> Active(file): 84 kB
> Inactive(file): 4184 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 0 kB
> Writeback: 0 kB
> AnonPages: 2556 kB
> Mapped: 1088 kB
> Shmem: 196 kB
> Slab: 509708 kB <------
> SReclaimable: 7596 kB <------
> SUnreclaim: 502112 kB <------
This indicates that your slab is not being reclaimed, but not why.
> KernelStack: 1096 kB
> PageTables: 748 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 6086632 kB
> Committed_AS: 9440 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 303488 kB
> VmallocChunk: 34359426132 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 6152 kB
> DirectMap2M: 2070528 kB
> DirectMap1G: 10485760 kB
>
> I look forward to hearing from you and thank you very much for your kind assistance.
Unfortunately I don't have solid answers for you at this point, nor a
solution. This is beyond my expertise. Maybe someone else with more
knowledge/experience will jump in. I suspect your application may be
doing something a bit unusual.
Cheers,
Stan
> Best Regards,
>
> Guochao
>
>
> At 2014-04-05 04:24:12,"Stan Hoeppner" <stan@hardwarefreak.com> wrote:
>> On 4/4/2014 2:26 AM, daiguochao wrote:
>>> Hello folks,
>>
>> Hello,
>>
>> Note that your problems are not XFS specific, but can occur with any
>> Linux filesystem.
>>
>>> I used xfs file system in kernel-2.6.32-220.13.1.el6.x86_64 for store
>>> pictures. About 100 days system memorys is lost and some nginx process is
>>> killed by oom-killer.So,I looked /proc/meminfo and find memorys is
>>> lost.Finally, I try to umount xfs system and 10GB memorys is coming back. l
>>> look xfs bugzilla no such BUG.I have no idea for it.
>>>
>>> Cheers,
>>>
>>> Guochao.
>>>
>>> some memorys info:
>>>
>>> 0> free -m
>>> total used free shared buffers cached
>>> Mem: 11887 11668 219 0 0 2
>>> -/+ buffers/cache: 11665 222
>>> Swap: 0 0 0
>>
>>
>> First problem: no swap
>> Second problem: cache is not being reclaimed
>>
>> Read vfs_cache_pressure at:
>> https://www.kernel.org/doc/Documentation/sysctl/vm.txt
>>
>> You've likely set this value to zero. Changing it to 200 should prompt
>> the kernel to reclaim dentries and inodes aggressively, preventing the
>> oom-killer from kicking in.
>>
>> Cheers,
>>
>> Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 10GB memorys occupied by XFS
2014-04-11 2:40 ` daiguochao
2014-04-11 4:26 ` Dave Chinner
@ 2014-04-11 21:35 ` Stan Hoeppner
1 sibling, 0 replies; 7+ messages in thread
From: Stan Hoeppner @ 2014-04-11 21:35 UTC (permalink / raw)
To: daiguochao, xfs
On 4/10/2014 9:40 PM, daiguochao wrote:
> Dear Stan, I can't send email to you.So I leave a message here.I hope not to
> bother you.
> Thank you for your kind assistance.
I received all of the ones you sent to the list and that should always
be the case. One that you sent directly to me was rejected but I think
I've fixed that now. And I think my delayed reply made things seem
worse than they are.
Anyway, Dave replied while I was typing my last response. He'll be much
more able to assist you. Your problem seems beyond the edge of my
knowledge.
Cheers,
Stan
> In accordance with your suggestion, we executed "echo 3 >
> /proc/sysm/drop_caches" for trying to release vfs dentries and inodes.
> Really,
> our lost memory came back. But we learned that the memory of vfs dentries
> and inodes is distributed from slab. Please check our system "Slab: 509708
> kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take
> up 450MB among. And /proc/meminfo indicated that our system memory is
> anomalous, there is about 10GB out of the statistics. We want to know how
> the system could observe the usage amount of vfs dentries and iodes through
> the system interface. If the memory usage of system is not reflected in
> /proc/meminfo as we can not find the statistics, and we thought it as a bug
> of xfs.
>
> My vm.vfs_cache_pressure of linux system is 100. We think that the system
> will proactively take the memory back when the memory is not enough, rather
> than oom-killer kills our work process. Our datas of /proc/meminfo occurred
> during the system problem as below:
> 130> cat /proc/meminfo
> MemTotal: 12173268 kB
> MemFree: 223044 kB
> Buffers: 244 kB
> Cached: 4540 kB
> SwapCached: 0 kB
> Active: 1700 kB
> Inactive: 5312 kB
> Active(anon): 1616 kB
> Inactive(anon): 1128 kB
> Active(file): 84 kB
> Inactive(file): 4184 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 0 kB
> Writeback: 0 kB
> AnonPages: 2556 kB
> Mapped: 1088 kB
> Shmem: 196 kB
> Slab: 509708 kB
> SReclaimable: 7596 kB
> SUnreclaim: 502112 kB
> KernelStack: 1096 kB
> PageTables: 748 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 6086632 kB
> Committed_AS: 9440 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 303488 kB
> VmallocChunk: 34359426132 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 6152 kB
> DirectMap2M: 2070528 kB
> DirectMap1G: 10485760 kB
>
> Best Regards,
>
> Guochao
>
>
>
> --
> View this message in context: http://xfs.9218.n7.nabble.com/10GB-memorys-occupied-by-XFS-tp35015p35016.html
> Sent from the Xfs - General mailing list archive at Nabble.com.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-04-11 21:35 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-04 7:26 10GB memorys occupied by XFS daiguochao
2014-04-04 20:24 ` Stan Hoeppner
[not found] ` <76016fc7.13c84.14546bad411.Coremail.dx-wl@163.com>
2014-04-10 1:41 ` 答复: " 戴国超
2014-04-11 5:09 ` Stan Hoeppner
2014-04-11 2:40 ` daiguochao
2014-04-11 4:26 ` Dave Chinner
2014-04-11 21:35 ` Stan Hoeppner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).