From: Olivier Bonvalet <xen.list@daevel.fr>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: xen-users@lists.xen.org, xen-devel@lists.xen.org
Subject: Re: [Xen-users] unexpected Out Of Memory (OOM)
Date: Wed, 07 Aug 2013 11:46:23 +0200 [thread overview]
Message-ID: <1375868783.13572.17.camel@localhost> (raw)
In-Reply-To: <1375868133.13572.13.camel@localhost>
Le mercredi 07 août 2013 à 11:35 +0200, Olivier Bonvalet a écrit :
> I have a DomU which is currently running with 8GB of memory (confirmed
> in /proc/meminfo), and only 6GB of memory is used, without process using
> it :
>
> root! to:~# free -m
> total used free shared buffers cached
> Mem: 8024 6642 1381 0 32 252
> -/+ buffers/cache: 6357 1667
> Swap: 255 0 255
>
>
>
> root! to:~# ps auxw
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 1 0.0 0.0 8408 800 ? Ss Aug06 0:01 init [2]
> root 2 0.0 0.0 0 0 ? S Aug06 0:00 [kthreadd]
> root 3 0.0 0.0 0 0 ? S Aug06 0:00 [ksoftirqd/0]
> root 4 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/0:0]
> root 5 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/0:0H]
> root 7 0.0 0.0 0 0 ? S Aug06 0:00 [migration/0]
> root 8 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_bh]
> root 9 0.0 0.0 0 0 ? S Aug06 0:00 [rcu_sched]
> root 10 0.0 0.0 0 0 ? S Aug06 0:00 [watchdog/0]
> root 11 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/1]
> root 12 0.0 0.0 0 0 ? R Aug06 0:00 [migration/1]
> root 13 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/1]
> root 14 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/1:0]
> root 15 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:0H]
> root 16 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/2]
> root 17 0.0 0.0 0 0 ? R Aug06 0:00 [migration/2]
> root 18 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/2]
> root 19 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/2:0]
> root 20 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:0H]
> root 21 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/3]
> root 22 0.0 0.0 0 0 ? R Aug06 0:00 [migration/3]
> root 23 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/3]
> root 24 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/3:0]
> root 25 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:0H]
> root 26 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/4]
> root 27 0.0 0.0 0 0 ? R Aug06 0:00 [migration/4]
> root 28 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/4]
> root 29 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/4:0]
> root 30 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:0H]
> root 31 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/5]
> root 32 0.0 0.0 0 0 ? R Aug06 0:00 [migration/5]
> root 33 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/5]
> root 34 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/5:0]
> root 35 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:0H]
> root 36 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/6]
> root 37 0.0 0.0 0 0 ? R Aug06 0:00 [migration/6]
> root 38 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/6]
> root 39 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/6:0]
> root 40 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:0H]
> root 41 0.0 0.0 0 0 ? R Aug06 0:00 [watchdog/7]
> root 42 0.0 0.0 0 0 ? R Aug06 0:00 [migration/7]
> root 43 0.0 0.0 0 0 ? R Aug06 0:00 [ksoftirqd/7]
> root 44 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/7:0]
> root 45 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:0H]
> root 46 0.0 0.0 0 0 ? S< Aug06 0:00 [khelper]
> root 47 0.0 0.0 0 0 ? S Aug06 0:00 [kdevtmpfs]
> root 48 0.0 0.0 0 0 ? S Aug06 0:00 [xenwatch]
> root 49 0.0 0.0 0 0 ? S Aug06 0:00 [xenbus]
> root 50 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/1:1H]
> root 51 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/2:1H]
> root 52 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/3:1H]
> root 53 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/4:1H]
> root 54 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/5:1H]
> root 55 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/6:1H]
> root 56 0.0 0.0 0 0 ? S< Aug06 0:00 [kworker/7:1H]
> root 57 0.0 0.0 0 0 ? S< Aug06 0:00 [writeback]
> root 58 0.0 0.0 0 0 ? S< Aug06 0:00 [kintegrityd]
> root 59 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
> root 60 0.0 0.0 0 0 ? S< Aug06 0:00 [kblockd]
> root 61 0.0 0.0 0 0 ? S< Aug06 0:00 [xenbus_frontend]
> root 62 0.0 0.0 0 0 ? S< Aug06 0:00 [ata_sff]
> root 63 0.0 0.0 0 0 ? S< Aug06 0:00 [edac-poller]
> root 64 0.0 0.0 0 0 ? S Aug06 0:02 [kworker/0:1]
> root 106 0.0 0.0 0 0 ? S Aug06 0:00 [kswapd0]
> root 107 0.0 0.0 0 0 ? SN Aug06 0:00 [ksmd]
> root 108 0.0 0.0 0 0 ? S Aug06 0:00 [fsnotify_mark]
> root 109 0.0 0.0 0 0 ? S< Aug06 0:00 [crypto]
> root 126 0.0 0.0 0 0 ? S Aug06 0:00 [khvcd]
> root 127 0.0 0.0 0 0 ? S< Aug06 0:00 [iscsi_eh]
> root 128 0.0 0.0 0 0 ? S< Aug06 0:00 [deferwq]
> root 129 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:1]
> root 253 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvda-8]
> root 254 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 257 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 259 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 520 0.0 0.0 0 0 ? S< Aug06 0:00 [kdmflush]
> root 704 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdc-8]
> root 706 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 707 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/xvdb-8]
> root 708 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 709 0.0 0.0 0 0 ? S Aug06 0:00 [jbd2/dm-0-8]
> root 710 0.0 0.0 0 0 ? S< Aug06 0:00 [ext4-dio-unwrit]
> root 713 0.0 0.0 0 0 ? S< Aug06 0:00 [bioset]
> root 714 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-worker-1]
> root 715 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-genwork-1]
> root 716 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-submit-1]
> root 717 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delalloc-]
> root 718 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-fixup-1]
> root 719 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-1]
> root 720 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
> root 721 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-rmw-1]
> root 722 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-rai]
> root 723 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-met]
> root 724 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-endio-wri]
> root 725 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-freespace]
> root 726 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-delayed-m]
> root 727 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cache-1]
> root 728 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-readahead]
> root 729 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-flush_del]
> root 730 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-qgroup-re]
> root 731 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-cleaner]
> root 732 0.0 0.0 0 0 ? S Aug06 0:00 [btrfs-transacti]
> root 1845 0.0 0.0 5984 620 tty1 Ss+ Aug06 0:00 /sbin/getty 38400 tty1
> root 1846 0.0 0.0 5984 624 hvc0 Ss+ Aug06 0:00 /sbin/getty 38400 hvc0
> root 1908 0.0 0.0 0 0 ? S Aug06 0:00 [kworker/u16:2]
> root 6457 0.0 0.0 76740 3348 ? Ss 11:23 0:00 sshd: daevel [priv]
> daevel 6460 0.0 0.0 76740 1928 ? S 11:23 0:00 sshd: daevel@pts/0,pts/1
> daevel 6461 0.0 0.0 25040 7312 pts/0 Ss 11:23 0:00 -bash
> daevel 6640 0.0 0.0 25036 7296 pts/1 Ss 11:23 0:00 -bash
> root 6668 0.0 0.0 25128 1264 pts/1 S 11:23 0:00 sudo su
> root 6669 0.0 0.0 45804 1348 pts/1 S 11:23 0:00 su
> root 6671 0.0 0.0 19944 2168 pts/1 S 11:23 0:00 bash
> daevel 8314 0.0 0.0 5924 560 pts/0 S+ 11:32 0:00 sleep 30
> root 8315 0.0 0.0 16900 1176 pts/1 R+ 11:33 0:00 ps auxw
> root 11919 0.0 0.0 0 0 ? S< 02:00 0:00 [kworker/0:1H]
> root 11971 0.0 0.0 0 0 ? S< 02:00 0:00 [bioset]
> root! to:~#
>
> root! to:~# cat /proc/meminfo
> MemTotal: 8217004 kB
> MemFree: 1415044 kB
> Buffers: 33128 kB
> Cached: 258884 kB
> SwapCached: 0 kB
> Active: 123328 kB
> Inactive: 182692 kB
> Active(anon): 14032 kB
> Inactive(anon): 344 kB
> Active(file): 109296 kB
> Inactive(file): 182348 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 262140 kB
> SwapFree: 262140 kB
> Dirty: 0 kB
> Writeback: 0 kB
> AnonPages: 14024 kB
> Mapped: 4300 kB
> Shmem: 396 kB
> Slab: 5105244 kB
> SReclaimable: 1563800 kB
> SUnreclaim: 3541444 kB
> KernelStack: 1360232 kB
> PageTables: 860 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 4370640 kB
> Committed_AS: 20612 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 16248 kB
> VmallocChunk: 34359721447 kB
> AnonHugePages: 0 kB
> DirectMap4k: 8388608 kB
> DirectMap2M: 0 kB
> root! to:~#
>
>
> root! to:~# slabtop --once ; sleep 30 ; echo ; slabtop --once
> Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
> Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
> Active / Total Caches (% used) : 72 / 93 (77.4%)
> Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
> Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
> 1997499 1997499 100% 0.19K 95119 21 380476K dentry
> 1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
> 1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
> 863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
> 180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
> 171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
> 170079 170079 100% 0.81K 4361 39 139552K task_xstate
> 170050 170050 100% 1.66K 8950 19 286400K task_struct
> 169860 169860 100% 1.06K 5662 30 181184K signal_cache
> 19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
> 19383 19383 100% 0.10K 497 39 1988K buffer_head
> 4608 4608 100% 0.01K 9 512 36K kmalloc-8
> 4592 4592 100% 0.55K 164 28 2624K radix_tree_node
> 4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
> 3660 3660 100% 0.53K 122 30 1952K inode_cache
> 3200 3200 100% 0.03K 25 128 100K kmalloc-32
> 2816 2816 100% 0.02K 11 256 44K kmalloc-16
> 2448 2448 100% 0.04K 24 102 96K ext4_extent_status
> 2112 2112 100% 0.06K 33 64 132K anon_vma
> 1632 1632 100% 0.04K 16 102 64K ext4_system_zone
> 1298 1186 91% 0.18K 59 22 236K vm_area_struct
> 1024 910 88% 0.50K 32 32 512K kmalloc-512
> 546 546 100% 0.09K 13 42 52K kmalloc-96
> 345 345 100% 0.34K 15 23 120K blkdev_requests
> 340 340 100% 0.05K 4 85 16K shared_policy_node
> 336 336 100% 0.19K 16 21 64K tw_sock_TCP
> 324 324 100% 0.11K 9 36 36K jbd2_journal_head
> 320 320 100% 1.00K 10 32 320K kmalloc-1024
> 312 312 100% 0.10K 8 39 32K flow_cache
> 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
> 232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
> 195 195 100% 2.06K 13 15 416K idr_layer_cache
> 195 179 91% 2.06K 13 15 416K sighand_cache
> 192 192 100% 0.06K 3 64 12K ext4_io_end
> 170 170 100% 0.02K 1 170 4K numa_policy
> 128 128 100% 0.06K 2 64 8K kmem_cache_node
> 128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
> 125 125 100% 0.62K 5 25 80K files_cache
> 125 125 100% 0.16K 5 25 20K sigqueue
> 120 80 66% 4.00K 15 8 480K kmalloc-4096
> 112 112 100% 2.00K 7 16 224K kmalloc-2048
> 100 100 100% 0.62K 4 25 64K sock_inode_cache
> 96 96 100% 0.25K 3 32 24K kmem_cache
> 96 96 100% 0.98K 3 32 96K btrfs_inode
>
> Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
> Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
> Active / Total Caches (% used) : 72 / 93 (77.4%)
> Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
> Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
> 1997520 1997520 100% 0.19K 95120 21 380480K dentry
> 1904890 1904890 100% 0.59K 73265 26 1172240K proc_inode_cache
> 1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
> 863488 861305 99% 0.06K 13492 64 53968K kmalloc-64
> 180540 180540 100% 0.11K 5015 36 20060K sysfs_dir_cache
> 171808 170841 99% 0.12K 5369 32 21476K kmalloc-128
> 170079 170079 100% 0.81K 4361 39 139552K task_xstate
> 170050 170050 100% 1.66K 8950 19 286400K task_struct
> 169860 169860 100% 1.06K 5662 30 181184K signal_cache
> 19925 19925 100% 0.62K 797 25 12752K shmem_inode_cache
> 19383 19383 100% 0.10K 497 39 1988K buffer_head
> 4608 4608 100% 0.01K 9 512 36K kmalloc-8
> 4592 4592 100% 0.55K 164 28 2624K radix_tree_node
> 4480 4480 100% 0.91K 128 35 4096K ext4_inode_cache
> 3660 3660 100% 0.53K 122 30 1952K inode_cache
> 3200 3200 100% 0.03K 25 128 100K kmalloc-32
> 2816 2816 100% 0.02K 11 256 44K kmalloc-16
> 2448 2448 100% 0.04K 24 102 96K ext4_extent_status
> 2112 2112 100% 0.06K 33 64 132K anon_vma
> 1632 1632 100% 0.04K 16 102 64K ext4_system_zone
> 1298 1186 91% 0.18K 59 22 236K vm_area_struct
> 1024 910 88% 0.50K 32 32 512K kmalloc-512
> 546 546 100% 0.09K 13 42 52K kmalloc-96
> 345 345 100% 0.34K 15 23 120K blkdev_requests
> 340 340 100% 0.05K 4 85 16K shared_policy_node
> 336 336 100% 0.19K 16 21 64K tw_sock_TCP
> 324 324 100% 0.11K 9 36 36K jbd2_journal_head
> 320 320 100% 1.00K 10 32 320K kmalloc-1024
> 312 312 100% 0.10K 8 39 32K flow_cache
> 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
> 232 89 38% 0.27K 8 29 64K nf_conntrack_ffffffff8165cc40
> 195 195 100% 2.06K 13 15 416K idr_layer_cache
> 195 179 91% 2.06K 13 15 416K sighand_cache
> 192 192 100% 0.06K 3 64 12K ext4_io_end
> 170 170 100% 0.02K 1 170 4K numa_policy
> 128 128 100% 0.06K 2 64 8K kmem_cache_node
> 128 128 100% 0.03K 1 128 4K jbd2_revoke_record_s
> 125 125 100% 0.62K 5 25 80K files_cache
> 125 125 100% 0.16K 5 25 20K sigqueue
> 120 80 66% 4.00K 15 8 480K kmalloc-4096
> 112 112 100% 2.00K 7 16 224K kmalloc-2048
> 100 100 100% 0.62K 4 25 64K sock_inode_cache
> 96 96 100% 0.25K 3 32 24K kmem_cache
> 96 96 100% 0.98K 3 32 96K btrfs_inode
> root! to:~#
>
>
>
>
and the diff between the two slaptop :
- Active / Total Objects (% used) : 17298069 / 17301644 (100.0%)
- Active / Total Slabs (% used) : 597110 / 597110 (100.0%)
+ Active / Total Objects (% used) : 17298196 / 17301771 (100.0%)
+ Active / Total Slabs (% used) : 597115 / 597115 (100.0%)
- Active / Total Size (% used) : 5045185.95K / 5045752.48K (100.0%)
+ Active / Total Size (% used) : 5045213.76K / 5045780.30K (100.0%)
-10290816 10290816 100% 0.25K 321588 32 2572704K kmalloc-256
-1997499 1997499 100% 0.19K 95119 21 380476K dentry
+10290880 10290880 100% 0.25K 321590 32 2572720K kmalloc-256
+1997520 1997520 100% 0.19K 95120 21 380480K dentry
-1305654 1305654 100% 0.19K 62174 21 248696K kmalloc-192
+1305696 1305696 100% 0.19K 62176 21 248704K kmalloc-192
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-08-07 9:46 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-07 0:02 unexpected Out Of Memory (OOM) Olivier Bonvalet
2013-08-07 8:29 ` [Xen-users] " Ian Campbell
2013-08-07 8:58 ` Olivier Bonvalet
2013-08-07 9:35 ` Olivier Bonvalet
2013-08-07 9:46 ` Olivier Bonvalet [this message]
2013-08-07 10:16 ` Ian Campbell
2013-08-07 11:17 ` Olivier Bonvalet
2013-08-07 13:36 ` Ian Campbell
2013-08-07 14:26 ` Olivier Bonvalet
2013-08-07 21:37 ` Olivier Bonvalet
2013-08-08 8:58 ` Ian Campbell
2013-08-08 10:10 ` Olivier Bonvalet
2013-08-08 10:18 ` Ian Campbell
2013-08-08 11:43 ` Olivier Bonvalet
2013-08-08 13:25 ` Wei Liu
2013-08-08 14:17 ` Olivier Bonvalet
2013-08-09 12:53 ` Konrad Rzeszutek Wilk
2013-08-07 14:48 ` Wei Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1375868783.13572.17.camel@localhost \
--to=xen.list@daevel.fr \
--cc=Ian.Campbell@citrix.com \
--cc=xen-devel@lists.xen.org \
--cc=xen-users@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.