From: Olivier Bonvalet <xen.list@daevel.fr>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: xen-users@lists.xen.org, xen-devel@lists.xen.org
Subject: Re: [Xen-users] unexpected Out Of Memory (OOM)
Date: Wed, 07 Aug 2013 11:35:33 +0200 [thread overview]
Message-ID: <1375868133.13572.13.camel@localhost> (raw)
In-Reply-To: <1375865894.13572.7.camel@localhost>
I have a DomU which is currently running with 8GB of memory (confirmed
in /proc/meminfo), and only 6GB of memory is used, without process using
it :
root! to:~# free -m
total used free shared buffers cached
Mem: 8024 6642 1381 0 32 252
-/+ buffers/cache: 6357 1667
Swap: 255 0 255
root! to:~# ps auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 8408 800 ? Ss Aug06 0:01 init [2]
root 2 0.0 0.0 0 0 ? S Aug06 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Aug06 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Aug06 0:00 [migration/0]
root 8 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_bh]
root 9 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_sched]
root 10 0.0 0.0 0 0 ? S Aug06 0:00 [watchdog/0]
root 11 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/1]
root 12 0.0 0.0 0 0 ? R Aug06 0:00 [migration/1]
root 13 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/1]
root 14 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/1:0]
root 15 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:0H]
root 16 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/2]
root 17 0.0 0.0 0 0 ? R Aug06 0:00 [migration/2]
root 18 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/2]
root 19 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/2:0]
root 20 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:0H]
root 21 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/3]
root 22 0.0 0.0 0 0 ? R Aug06 0:00 [migration/3]
root 23 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/3]
root 24 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/3:0]
root 25 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:0H]
root 26 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/4]
root 27 0.0 0.0 0 0 ? R Aug06 0:00 [migration/4]
root 28 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/4]
root 29 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/4:0]
root 30 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:0H]
root 31 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/5]
root 32 0.0 0.0 0 0 ? R Aug06 0:00 [migration/5]
root 33 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/5]
root 34 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/5:0]
root 35 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:0H]
root 36 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/6]
root 37 0.0 0.0 0 0 ? R Aug06 0:00 [migration/6]
root 38 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/6]
root 39 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/6:0]
root 40 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:0H]
root 41 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/7]
root 42 0.0 0.0 0 0 ? R Aug06 0:00 [migration/7]
root 43 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/7]
root 44 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/7:0]
root 45 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:0H]
root 46 0.0 0.0 0 0 ? S< Aug06 0:00 [khelper]
root 47 0.0 0.0 0 0 ? S Aug06 0:00 [kdevtmpfs]
root 48 0.0 0.0 0 0 ? S Aug06 0:00 [xenwatch]
root 49 0.0 0.0 0 0 ? S Aug06 0:00 [xenbus]
root 50 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:1H]
root 51 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:1H]
root 52 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:1H]
root 53 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:1H]
root 54 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:1H]
root 55 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:1H]
root 56 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:1H]
root 57 0.0 0.0 0 0 ? S< Aug06 0:00 [writeback]
root 58 0.0 0.0 0 0 ? S< Aug06 0:00 [kintegrityd]
root 59 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
root 60 0.0 0.0 0 0 ? S< Aug06 0:00 [kblockd]
root 61 0.0 0.0 0 0 ? S< Aug06 0:00 [xenbus_frontend]
root 62 0.0 0.0 0 0 ? S< Aug06 0:00 [ata_sff]
root 63 0.0 0.0 0 0 ? S< Aug06 0:00 [edac-poller]
root 64 0.0 0.0 0 0 ? S Aug06 0:02 [kworker/0:1]
root 106 0.0 0.0 0 0 ? S Aug06 0:00 [kswapd0]
root 107 0.0 0.0 0 0 ? SN Aug06 0:00 [ksmd]
root 108 0.0 0.0 0 0 ? S Aug06 0:00 [fsnotify_mark]
root 109 0.0 0.0 0 0 ? S< Aug06 0:00 [crypto]
root 126 0.0 0.0 0 0 ? S Aug06 0:00 [khvcd]
root 127 0.0 0.0 0 0 ? S< Aug06 0:00 [iscsi_eh]
root 128 0.0 0.0 0 0 ? S< Aug06 0:00 [deferwq]
root 129 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:1]
root 253 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvda-8]
root 254 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 257 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 259 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 520 0.0 0.0 0 0 ? S< Aug06 0:00 [kdmflush]
root 704 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdc-8]
root 706 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 707 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdb-8]
root 708 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 709 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/dm-0-8]
root 710 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
root 713 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
root 714 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-worker-1]
root 715 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-genwork-1]
root 716 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-submit-1]
root 717 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delalloc-]
root 718 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-fixup-1]
root 719 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-1]
root 720 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
root 721 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-rmw-1]
root 722 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-rai]
root 723 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
root 724 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-wri]
root 725 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-freespace]
root 726 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delayed-m]
root 727 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cache-1]
root 728 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-readahead]
root 729 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-flush_del]
root 730 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-qgroup-re]
root 731 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cleaner]
root 732 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-transacti]
root 1845 0.0 0.0 5984 620 tty1 Ss+ Aug06 0:00 /sbin/getty 38400 tty1
root 1846 0.0 0.0 5984 624 hvc0 Ss+ Aug06 0:00 /sbin/getty 38400 hvc0
root 1908 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:2]
root 6457 0.0 0.0 76740 3348 ? Ss 11:23 0:00 sshd: daevel [priv]
daevel 6460 0.0 0.0 76740 1928 ? S 11:23 0:00 sshd: daevel@pts/0,pts/1
daevel 6461 0.0 0.0 25040 7312 pts/0 Ss 11:23 0:00 -bash
daevel 6640 0.0 0.0 25036 7296 pts/1 Ss 11:23 0:00 -bash
root 6668 0.0 0.0 25128 1264 pts/1 S 11:23 0:00 sudo su
root 6669 0.0 0.0 45804 1348 pts/1 S 11:23 0:00 su
root 6671 0.0 0.0 19944 2168 pts/1 S 11:23 0:00 bash
daevel 8314 0.0 0.0 5924 560 pts/0 S+ 11:32 0:00 sleep 30
root 8315 0.0 0.0 16900 1176 pts/1 R+ 11:33 0:00 ps auxw
root 11919 0.0 0.0 0 0 ? S< 02:00 0:00 [kworker/0:1H]
root 11971 0.0 0.0 0 0 ? S< 02:00 0:00 [bioset]
root! to:~#
root! to:~# cat /proc/meminfo
MemTotal: 8217004 kB
MemFree: 1415044 kB
Buffers: 33128 kB
Cached: 258884 kB
SwapCached: 0 kB
Active: 123328 kB
Inactive: 182692 kB
Active(anon): 14032 kB
Inactive(anon): 344 kB
Active(file): 109296 kB
Inactive(file): 182348 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 262140 kB
SwapFree: 262140 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 14024 kB
Mapped: 4300 kB
Shmem: 396 kB
Slab: 5105244 kB
SReclaimable: 1563800 kB
SUnreclaim: 3541444 kB
KernelStack: 1360232 kB
PageTables: 860 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4370640 kB
Committed_AS: 20612 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 16248 kB
VmallocChunk: 34359721447 kB
AnonHugePages: 0 kB
DirectMap4k: 8388608 kB
DirectMap2M: 0 kB
root! to:~#
root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
Active / Total Caches (% used) : 72 / 93 (77.4%)
Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
1997499 1997499 100% 0.19K 95119 21 380476K dentry
1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
170079 170079 100% 0.81K 4361 39 139552K task_xstate
170050 170050 100% 1.66K 8950 19 286400K task_struct
169860 169860 100% 1.06K 5662 30 181184K signal_cache
19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
19383 19383 100% 0.10K 497 39 1988K buffer_head
4608 4608 100% 0.01K 9 512 36K kmalloc-8
4592 4592 100% 0.55K 164 28 2624K radix_tree_node
4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
3660 3660 100% 0.53K 122 30 1952K inode_cache
3200 3200 100% 0.03K 25 128 100K kmalloc-32
2816 2816 100% 0.02K 11 256 44K kmalloc-16
2448 2448 100% 0.04K 24 102 96K ext4_extent_status
2112 2112 100% 0.06K 33 64 132K anon_vma
1632 1632 100% 0.04K 16 102 64K ext4_system_zone
1298 1186 91% 0.18K 59 22 236K vm_area_struct
1024 910 88% 0.50K 32 32 512K kmalloc-512
546 546 100% 0.09K 13 42 52K kmalloc-96
345 345 100% 0.34K 15 23 120K blkdev_requests
340 340 100% 0.05K 4 85 16K shared_policy_node
336 336 100% 0.19K 16 21 64K tw_sock_TCP
324 324 100% 0.11K 9 36 36K jbd2_journal_head
320 320 100% 1.00K 10 32 320K kmalloc-1024
312 312 100% 0.10K 8 39 32K flow_cache
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
195 195 100% 2.06K 13 15 416K idr_layer_cache
195 179 91% 2.06K 13 15 416K sighand_cache
192 192 100% 0.06K 3 64 12K ext4_io_end
170 170 100% 0.02K 1 170 4K numa_policy
128 128 100% 0.06K 2 64 8K kmem_cache_node
128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
125 125 100% 0.62K 5 25 80K files_cache
125 125 100% 0.16K 5 25 20K sigqueue
120 80 66% 4.00K 15 8 480K kmalloc-4096
112 112 100% 2.00K 7 16 224K kmalloc-2048
100 100 100% 0.62K 4 25 64K sock_inode_cache
96 96 100% 0.25K 3 32 24K kmem_cache
96 96 100% 0.98K 3 32 96K btrfs_inode
Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
Active / Total Caches (% used) : 72 / 93 (77.4%)
Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
1997520 1997520 100% 0.19K 95120 21 380480K dentry
1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
170079 170079 100% 0.81K 4361 39 139552K task_xstate
170050 170050 100% 1.66K 8950 19 286400K task_struct
169860 169860 100% 1.06K 5662 30 181184K signal_cache
19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
19383 19383 100% 0.10K 497 39 1988K buffer_head
4608 4608 100% 0.01K 9 512 36K kmalloc-8
4592 4592 100% 0.55K 164 28 2624K radix_tree_node
4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
3660 3660 100% 0.53K 122 30 1952K inode_cache
3200 3200 100% 0.03K 25 128 100K kmalloc-32
2816 2816 100% 0.02K 11 256 44K kmalloc-16
2448 2448 100% 0.04K 24 102 96K ext4_extent_status
2112 2112 100% 0.06K 33 64 132K anon_vma
1632 1632 100% 0.04K 16 102 64K ext4_system_zone
1298 1186 91% 0.18K 59 22 236K vm_area_struct
1024 910 88% 0.50K 32 32 512K kmalloc-512
546 546 100% 0.09K 13 42 52K kmalloc-96
345 345 100% 0.34K 15 23 120K blkdev_requests
340 340 100% 0.05K 4 85 16K shared_policy_node
336 336 100% 0.19K 16 21 64K tw_sock_TCP
324 324 100% 0.11K 9 36 36K jbd2_journal_head
320 320 100% 1.00K 10 32 320K kmalloc-1024
312 312 100% 0.10K 8 39 32K flow_cache
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
195 195 100% 2.06K 13 15 416K idr_layer_cache
195 179 91% 2.06K 13 15 416K sighand_cache
192 192 100% 0.06K 3 64 12K ext4_io_end
170 170 100% 0.02K 1 170 4K numa_policy
128 128 100% 0.06K 2 64 8K kmem_cache_node
128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
125 125 100% 0.62K 5 25 80K files_cache
125 125 100% 0.16K 5 25 20K sigqueue
120 80 66% 4.00K 15 8 480K kmalloc-4096
112 112 100% 2.00K 7 16 224K kmalloc-2048
100 100 100% 0.62K 4 25 64K sock_inode_cache
96 96 100% 0.25K 3 32 24K kmem_cache
96 96 100% 0.98K 3 32 96K btrfs_inode
root! to:~#
next prev parent reply other threads:[~2013-08-07 9:35 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-07 0:02 unexpected Out Of Memory (OOM) Olivier Bonvalet
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
2013-08-07 8:58 ` Olivier Bonvalet
2013-08-07 9:35 ` Olivier Bonvalet [this message]
2013-08-07 9:46 ` Olivier Bonvalet
2013-08-07 10:16 ` Ian Campbell
2013-08-07 11:17 ` Olivier Bonvalet
2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
2013-08-08 10:10 ` Olivier Bonvalet
2013-08-08 10:18 ` Ian Campbell
2013-08-08 11:43 ` Olivier Bonvalet
2013-08-08 13:25 ` Wei Liu
2013-08-08 14:17 ` Olivier Bonvalet
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
2013-08-07 14:48 ` Wei Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1375868133.13572.13.camel@localhost \
--to=xen.list@daevel.fr \
--cc=Ian.Campbell@citrix.com \
--cc=xen-devel@lists.xen.org \
--cc=xen-users@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.