Re: Bug report - OOM killer kills task outside of cgroup

[Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>]

(cc'ing memcg maintainers and quoting whole body)

On Thu, Sep 11, 2014 at 02:05:19PM +1200, Tyler Power wrote:
> Hi there,
> 
> Hopefully I'm sending this to the right place, this is the first time
> I've reported a kernel bug. I'm roughly following this format here
> https://www.kernel.org/pub/linux/docs/lkml/reporting-bugs.html.
> 
> 1. The OOM killer kicks in to kill processes inside a cgroup that has
> hit its memory limit but sometimes kills a process outside of the
> cgroup
> 
> 2. We've encountered an error on Ubuntu 12.04 running on vsphere with
> kernel linux-image-3.13.0-32-generic as well as
> linux-image-3.13.0-35-generic which causes the machine to hard lock
> up. It is completely unresponsive until hard reset.
> 
> The issue appears to be that the OOM killer kicks in and kills a
> process outside of the cgroup that has hit its memory limit. Once this
> has happened the kernel log shows soft lock ups and eventually
> completely hangs.
> 
> We can fairly consistently reproduce this by starting some processes
> inside an LXC container (via docker) that push the container very
> close to its memory limit. We then have a cron task that starts up
> inside the container and causes it to OOM. The OOM killer then kicks
> in and sometimes will trigger this hang.
> 
> Every time it happens we see this message:
> 
> [ 2635.212105] Task in / killed as a result of limit of
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> 
> Notice how it kills a task in "/"? Immediately following that we start
> seeing this repeated:
> 
> [ 3164.153296] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940]
> 
> And then the machine hangs.
> 
> It looks very similar to
> http://permalink.gmane.org/gmane.linux.kernel.mm/108113 and
> http://irc.13thfloor.at/LOG/2014-02/LOG_2014-02-28.txt +
> http://217.196.41.9/~ard/bertl-is-the-nicest-guy-in-the-world/jimmy.txt
> (search for "in / killed").
> 
> Kernel log excerpt (fetched via serial using
> GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 debug
> ignore_loglevel nomodeset pciehp_force=1 pciehp_poll_mode=1")
> 
> [ 2395.454778] docker0: port 2(vethdRqyGV) entered forwarding state
> [ 2410.500706] docker0: port 2(vethdRqyGV) entered forwarding state
> [ 2634.814382] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> [ 2634.818286] java
> cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> mems_allowed=0
> [ 2634.822596] CPU: 1 PID: 15775 Comm: java Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2634.824960] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2634.828336]  ffff8801375a2000 ffff880137f0bbe8 ffffffff81752c9e
> ffff88013fd0fff0
> [ 2634.830662]  ffff8800ab6ddfc0 ffff880137f0bc38 ffffffff81748aba
> ffff880100000000
> [ 2634.832972]  000000d08137cd48 ffff880137f0bc38 ffff8800ae5ec7d0
> 0000000000000000
> [ 2634.835253] Call Trace:
> [ 2634.836418]  [<ffffffff81752c9e>] dump_stack+0x46/0x58
> [ 2634.838078]  [<ffffffff81748aba>] dump_header+0x7e/0xbd
> [ 2634.839825]  [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4
> [ 2634.841768]  [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20
> [ 2634.843776]  [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140
> [ 2634.845637]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2634.847424]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2634.849413]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2634.851413]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2634.853296]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2634.855197]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2634.857020]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2634.858815]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2634.860733]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2634.862595]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2634.864476]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2634.866197]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2634.867954] Task in
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> killed as a result of limit of
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> [ 2634.872851] memory: usage 65536kB, limit 65536kB, failcnt 34251
> [ 2634.874714] memory+swap: usage 65536kB, limit 65536kB, failcnt 1
> [ 2634.876622] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> [ 2634.878527] Memory cgroup stats for
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4:
> cache:412KB rss:65124KB rss_huge:0KB mapped_file:4KB writeback:0KB
> swap:0KB inactive_anon:33392KB active_anon:32052KB inactive_file:88KB
> active_file:0KB unevictable:0KB
> [ 2634.888650] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
> oom_score_adj name
> [ 2634.891018] [15552]     0 15552    12511      732      29        0
>            0 sshd
> [ 2634.893373] [15596]     0 15596     5180      276      15        0
>            0 cron
> [ 2634.895686] [15731]     0 15731    19971      926      43        0
>            0 sshd
> [ 2634.898004] [15735]  1014 15735    19971      395      40        0
>            0 sshd
> [ 2634.900305] [15736]  1014 15736     2760      376      10        0
>            0 bash
> [ 2634.902588] [15756]  1014 15756   551397    14730      92        0
>            0 java
> [ 2634.904853] [15757]  1014 15757     2760      152      10        0
>            0 bash
> [ 2634.907080] [15758]  1014 15758     2760      158      10        0
>            0 bash
> [ 2634.909316] [15759]  1014 15759     1472      171       7        0
>            0 tee
> [ 2634.911495] [15760]  1014 15760     1472      172       8        0
>            0 tee
> [ 2634.913689] [15936]     0 15936    11535      338      28        0
>            0 cron
> [ 2634.915905] [15937]     0 15937     1102      153       8        0
>            0 sh
> [ 2634.918055] [15938]     0 15938     1102      153       8        0
>            0 maxlifetime
> [ 2634.920385] [15940]     0 15940    53661     2029     105        0
>            0 php5
> [ 2634.922570] Memory cgroup out of memory: Kill process 15919 (java)
> score 904 or sacrifice child
> [ 2634.924952] Killed process 15758 (bash) total-vm:11040kB,
> anon-rss:216kB, file-rss:416kB
> [ 2634.937740] php5 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> [ 2634.939903] php5
> cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> mems_allowed=0
> [ 2634.943244] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2634.945628] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2634.949166]  ffff8801375a2000 ffff8800920c5be8 ffffffff81752c9e
> ffff88013fc0fff0
> [ 2634.951526]  ffff8800ab1a47d0 ffff8800920c5c38 ffffffff81748aba
> ffff880000000000
> [ 2634.953906]  000000d08137cd48 ffff8800920c5c38 ffff8800ae5ec7d0
> 0000000000000000
> [ 2634.956274] Call Trace:
> [ 2634.957475]  [<ffffffff81752c9e>] dump_stack+0x46/0x58
> [ 2634.959212]  [<ffffffff81748aba>] dump_header+0x7e/0xbd
> [ 2634.960979]  [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4
> [ 2634.962970]  [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20
> [ 2634.965016]  [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140
> [ 2634.966913]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2634.968739]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2634.970768]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2634.972788]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2634.974613]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2634.976509]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2634.978233]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2634.980032]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2634.981886]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2634.983713]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2634.985577]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2634.987283]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2634.988982] Task in
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> killed as a result of limit of
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> [ 2634.993837] memory: usage 65536kB, limit 65536kB, failcnt 34462
> [ 2634.995715] memory+swap: usage 65536kB, limit 65536kB, failcnt 1
> [ 2634.997667] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> [ 2634.999596] Memory cgroup stats for
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4:
> cache:464KB rss:65032KB rss_huge:0KB mapped_file:16KB writeback:0KB
> swap:0KB inactive_anon:33288KB active_anon:32068KB inactive_file:92KB
> active_file:12KB unevictable:0KB
> [ 2635.008960] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
> oom_score_adj name
> [ 2635.011353] [15552]     0 15552    12511      732      29        0
>            0 sshd
> [ 2635.013738] [15596]     0 15596     5180      276      15        0
>            0 cron
> [ 2635.016195] [15731]     0 15731    19971      926      43        0
>            0 sshd
> [ 2635.018559] [15735]  1014 15735    19971      395      40        0
>            0 sshd
> [ 2635.020947] [15736]  1014 15736     2760      376      10        0
>            0 bash
> [ 2635.023288] [15756]  1014 15756   551397    14759      92        0
>            0 java
> [ 2635.025590] [15757]  1014 15757     2760      152      10        0
>            0 bash
> [ 2635.027821] [15759]  1014 15759     1472      171       7        0
>            0 tee
> [ 2635.030055] [15760]  1014 15760     1472      172       8        0
>            0 tee
> [ 2635.032287] [15936]     0 15936    11535      338      28        0
>            0 cron
> [ 2635.034570] [15937]     0 15937     1102      153       8        0
>            0 sh
> [ 2635.036809] [15938]     0 15938     1102      153       8        0
>            0 maxlifetime
> [ 2635.039211] [15940]     0 15940    53661     2029     105        0
>            0 php5
> [ 2635.041469] Memory cgroup out of memory: Kill process 15919 (java)
> score 904 or sacrifice child
> [ 2635.043872] Killed process 15757 (bash) total-vm:11040kB,
> anon-rss:216kB, file-rss:392kB
> [ 2635.050025] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> [ 2635.052155] java
> cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> mems_allowed=0
> [ 2635.055293] CPU: 0 PID: 15768 Comm: java Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2635.057673] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2635.061178]  ffff8801375a2000 ffff88009212fbe8 ffffffff81752c9e
> 000000000000b9e4
> [ 2635.063548]  ffff8800b858dfc0 ffff88009212fc38 ffffffff81748aba
> ffff880000000000
> [ 2635.066421]  000000d08137cd48 ffff88009212fc38 ffff8800ae5ec7d0
> 0000000000000000
> [ 2635.069043] Call Trace:
> [ 2635.070234]  [<ffffffff81752c9e>] dump_stack+0x46/0x58
> [ 2635.072018]  [<ffffffff81748aba>] dump_header+0x7e/0xbd
> [ 2635.073804]  [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4
> [ 2635.075796]  [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20
> [ 2635.077847]  [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140
> [ 2635.079754]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2635.081600]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2635.083604]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2635.085649]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2635.087488]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2635.089409]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2635.091155]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2635.092958]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2635.094819]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2635.096657]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2635.098493]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2635.100257]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2635.101917] Task in
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> killed as a result of limit of
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> [ 2635.106727] memory: usage 65536kB, limit 65536kB, failcnt 34826
> [ 2635.108694] memory+swap: usage 65416kB, limit 65536kB, failcnt 1
> [ 2635.110609] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> [ 2635.112573] Memory cgroup stats for
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4:
> cache:452KB rss:65084KB rss_huge:0KB mapped_file:4KB writeback:0KB
> swap:0KB inactive_anon:33304KB active_anon:32104KB inactive_file:128KB
> active_file:0KB unevictable:0KB
> [ 2635.120953] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
> oom_score_adj name
> [ 2635.123326] [15552]     0 15552    12511      732      29        0
>            0 sshd
> [ 2635.125696] [15596]     0 15596     5180      276      15        0
>            0 cron
> [ 2635.128042] [15731]     0 15731    19971      926      43        0
>            0 sshd
> [ 2635.130337] [15735]  1014 15735    19971      395      40        0
>            0 sshd
> [ 2635.132641] [15736]  1014 15736     2760      376      10        0
>            0 bash
> [ 2635.134904] [15756]  1014 15756   551397    14752      92        0
>            0 java
> [ 2635.137170] [15759]  1014 15759     1472      171       7        0
>            0 tee
> [ 2635.139389] [15760]  1014 15760     1472      172       8        0
>            0 tee
> [ 2635.141612] [15936]     0 15936    11535      338      28        0
>            0 cron
> [ 2635.143833] [15937]     0 15937     1102      153       8        0
>            0 sh
> [ 2635.146024] [15938]     0 15938     1102      153       8        0
>            0 maxlifetime
> [ 2635.148378] [15940]     0 15940    55252     2162     108        0
>            0 php5
> [ 2635.150580] Memory cgroup out of memory: Kill process 15919 (java)
> score 906 or sacrifice child
> [ 2635.153010] Killed process 15919 (java) total-vm:2205588kB,
> anon-rss:58444kB, file-rss:564kB
> [ 2635.160153] php5 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> [ 2635.162238] php5
> cpuset=e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> mems_allowed=0
> [ 2635.165420] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2635.167811] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2635.171392]  ffff8801375a2000 ffff8800920c5be8 ffffffff81752c9e
> 0000000000000007
> [ 2635.173904]  ffff8800ab1a47d0 ffff8800920c5c38 ffffffff81748aba
> ffff880000000000
> [ 2635.176363]  000000d08137cd48 ffff8800920c5c38 ffff880137f2c7d0
> 0000000000000000
> [ 2635.178731] Call Trace:
> [ 2635.179924]  [<ffffffff81752c9e>] dump_stack+0x46/0x58
> [ 2635.181732]  [<ffffffff81748aba>] dump_header+0x7e/0xbd
> [ 2635.183532]  [<ffffffff81748b50>] oom_kill_process.part.5+0x57/0x2d4
> [ 2635.185606]  [<ffffffff810751b5>] ? has_ns_capability_noaudit+0x15/0x20
> [ 2635.187661]  [<ffffffff8115b499>] ? oom_badness.part.4+0xa9/0x140
> [ 2635.189642]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2635.191472]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2635.193567]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2635.195601]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2635.197548]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2635.199548]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2635.201309]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2635.203099]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2635.205016]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2635.206847]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2635.208699]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2635.210413]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> **-> [ 2635.212105] Task in / killed as a result of limit of
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4
> [ 2635.215651] memory: usage 5400kB, limit 65536kB, failcnt 34938
> [ 2635.217580] memory+swap: usage 5400kB, limit 65536kB, failcnt 1
> [ 2635.219498] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> [ 2635.221488] Memory cgroup stats for
> /lxc/e177098cd5f95ff8dbfa1ea14667b0bdc525dfa2e1c1b3bf763acd0a7ef217a4:
> cache:452KB rss:4088KB rss_huge:0KB mapped_file:0KB writeback:0KB
> swap:0KB inactive_anon:1080KB active_anon:3328KB inactive_file:132KB
> active_file:0KB unevictable:0KB
> [ 2635.233441] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents
> oom_score_adj name
> [ 2635.235809] [15552]     0 15552    12511      732      29        0
>            0 sshd
> [ 2635.238209] [15596]     0 15596     5180      276      15        0
>            0 cron
> [ 2635.240604] [15936]     0 15936    11535      338      28        0
>            0 cron
> [ 2635.242918] [15937]     0 15937     1102      153       8        0
>            0 sh
> [ 2635.245179] [15938]     0 15938     1102      153       8        0
>            0 maxlifetime
> [ 2635.247564] [15940]     0 15940    55252     2162     108        0
>            0 php5
> [ 2635.249819] Memory cgroup out of memory: Kill process 15861 (java)
> score 918 or sacrifice child
> [ 2659.884484] BUG: soft lockup - CPU#0 stuck for 23s! [php5:15940]
> [ 2659.888830] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp
> xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx
> serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp
> i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase
> vmw_pvscsi vmxnet3
> [ 2659.908560] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2659.912330] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2659.917958] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti:
> ffff8800920c4000
> [ 2659.921601] RIP: 0010:[<ffffffff81748c29>]  [<ffffffff81748c29>]
> oom_kill_process.part.5+0x130/0x2d4
> [ 2659.927119] RSP: 0000:ffff8800920c5c48  EFLAGS: 00000246
> [ 2659.930090] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000
> [ 2659.932371] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040
> [ 2659.934649] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410
> [ 2660.093701] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0
> [ 2660.097119] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000
> [ 2660.100528] FS:  00007ff405dc2700(0000) GS:ffff88013fc00000(0000)
> knlGS:0000000000000000
> [ 2660.103897] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2660.105903] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0
> [ 2660.108262] Stack:
> [ 2660.109477]  ffff8800920c5c58 ffff880137f2c7d0 0000000000004000
> ffff8801375a2000
> [ 2660.111952]  ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000
> ffff880137f2c7d0
> [ 2660.114437]  0000000000004000 0000000000000000 ffff8801375a2000
> 0000000000000000
> [ 2660.116912] Call Trace:
> [ 2660.118209]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2660.120166]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2660.122291]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2660.124439]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2660.126415]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2660.128503]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2660.130405]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2660.132395]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2660.134474]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2660.136605]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2660.138612]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2660.140483]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2660.364234] Code: 4d 8b a4 24 18 03 00 00 49 81 ec 18 03 00 00 49
> 8d 84 24 18 03 00 00 4c 39 e8 75 9c 49 8b 87 98 03 00 00 48 89 45 c8
> 4c 8b 7d c8 <49> 81 ef 98 03 00 00 4c 39 fb 0f 85 66 ff ff ff f0 ff 05
> 00 d4
> [ 2687.763079] BUG: soft lockup - CPU#0 stuck for 23s! [php5:15940]
> [ 2687.765130] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp
> xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx
> serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp
> i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase
> vmw_pvscsi vmxnet3
> [ 2687.778062] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2687.781178] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2687.784908] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti:
> ffff8800920c4000
> [ 2687.787208] RIP: 0010:[<ffffffff81748b9f>]  [<ffffffff81748b9f>]
> oom_kill_process.part.5+0xa6/0x2d4
> [ 2687.790638] RSP: 0000:ffff8800920c5c48  EFLAGS: 00000202
> [ 2687.792506] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000
> [ 2687.794825] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040
> [ 2687.797074] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410
> [ 2687.799314] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0
> [ 2687.801565] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000
> [ 2688.028141] FS:  00007ff405dc2700(0000) GS:ffff88013fc00000(0000)
> knlGS:0000000000000000
> [ 2688.031808] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2688.034737] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0
> [ 2688.037416] Stack:
> [ 2688.038594]  ffff8800920c5c58 ffff880137f2c7d0 0000000000004000
> ffff8801375a2000
> [ 2688.041050]  ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000
> ffff880137f2c7d0
> [ 2688.043501]  0000000000004000 0000000000000000 ffff8801375a2000
> 0000000000000000
> [ 2688.045972] Call Trace:
> [ 2688.047262]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2688.049245]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2688.051383]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2688.053562]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2688.055578]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2688.057675]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2688.059606]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2688.061588]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2688.063655]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2688.065647]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2688.067645]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2688.069508]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2688.071297] Code: 00 00 45 89 e8 48 c7 c7 08 50 a7 81 31 c0 e8 1c
> e1 ff ff 4c 89 e7 e8 b1 66 01 00 48 c7 c7 40 60 c0 81 e8 05 63 01 00
> 48 89 5d a8 <4d> 8b a7 08 03 00 00 4d 8d af 08 03 00 00 49 81 ec 18 03
> 00 00
> [ 2715.786036] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940]
> [ 2715.788772] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp
> xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx
> serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp
> i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase
> vmw_pvscsi vmxnet3
> [ 2715.801184] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2715.803616] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2715.807270] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti:
> ffff8800920c4000
> [ 2715.809571] RIP: 0010:[<ffffffff81748c30>]  [<ffffffff81748c30>]
> oom_kill_process.part.5+0x137/0x2d4
> [ 2715.812972] RSP: 0000:ffff8800920c5c48  EFLAGS: 00000286
> [ 2715.814858] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000
> [ 2715.817102] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040
> [ 2715.819350] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410
> [ 2715.821619] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0
> [ 2715.823852] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000
> [ 2715.826095] FS:  00007ff405dc2700(0000) GS:ffff88013fc00000(0000)
> knlGS:0000000000000000
> [ 2716.058770] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2716.060776] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0
> [ 2716.063102] Stack:
> [ 2716.064279]  ffff8800920c5c58 ffff880137f2c7d0 0000000000004000
> ffff8801375a2000
> [ 2716.066731]  ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000
> ffff880137f2c7d0
> [ 2716.069221]  0000000000004000 0000000000000000 ffff8801375a2000
> 0000000000000000
> [ 2716.071694] Call Trace:
> [ 2716.072983]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2716.074951]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2716.077116]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2716.079305]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2716.081322]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2716.083443]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2716.085375]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2716.087364]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2716.089458]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2716.091451]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2716.093447]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2716.095312]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2716.097125] Code: 00 49 81 ec 18 03 00 00 49 8d 84 24 18 03 00 00
> 4c 39 e8 75 9c 49 8b 87 98 03 00 00 48 89 45 c8 4c 8b 7d c8 49 81 ef
> 98 03 00 00 <4c> 39 fb 0f 85 66 ff ff ff f0 ff 05 00 d4 4b 00 48 8b 7d
> a8 e8
> [ 2743.821209] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940]
> [ 2743.823189] Modules linked in: xt_nat veth ipt_REJECT xt_tcpudp
> xt_state xt_addrtype iptable_filter xt_conntrack ipt_MASQUERADE
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack ip_tables x_tables bridge stp llc aufs psmouse vmwgfx
> serio_raw ttm drm ppdev vmw_vmci vmw_balloon parport_pc shpchp
> i2c_piix4 mac_hid lp parport e1000 floppy mptspi mptscsih mptbase
> vmw_pvscsi vmxnet3
> [ 2743.835431] CPU: 0 PID: 15940 Comm: php5 Not tainted
> 3.13.0-32-generic #57~precise1-Ubuntu
> [ 2743.837879] Hardware name: VMware, Inc. VMware Virtual
> Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
> [ 2743.841550] task: ffff8800ab1a47d0 ti: ffff8800920c4000 task.ti:
> ffff8800920c4000
> [ 2743.843848] RIP: 0010:[<ffffffff81748ba6>]  [<ffffffff81748ba6>]
> oom_kill_process.part.5+0xad/0x2d4
> [ 2743.847267] RSP: 0000:ffff8800920c5c48  EFLAGS: 00000202
> [ 2743.849129] RAX: ffff8800a0ffe358 RBX: 0000000000000396 RCX: 0000000000000000
> [ 2743.851396] RDX: ffff88013fc0fff0 RSI: ffff88013fc0e3d8 RDI: ffffffff81c06040
> [ 2743.974848] RBP: ffff8800920c5ca8 R08: 0000000000000000 R09: ffff8800ba7b1410
> [ 2743.978217] R10: 0000000000000773 R11: ffffffff8185b248 R12: ffff880137f2c7d0
> [ 2743.981510] R13: 0000000000000005 R14: 0000000000000246 R15: 00000000375a2000
> [ 2743.983737] FS:  00007ff405dc2700(0000) GS:ffff88013fc00000(0000)
> knlGS:0000000000000000
> [ 2743.986172] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 2743.988121] CR2: 000000000105c188 CR3: 00000000b86e1000 CR4: 00000000000007f0
> [ 2743.990371] Stack:
> [ 2743.991555]  ffff8800920c5c58 ffff880137f2c7d0 0000000000004000
> ffff8801375a2000
> [ 2743.994003]  ffff8800920c5ca8 ffff8800a0ffe358 ffff8800ba578000
> ffff880137f2c7d0
> [ 2743.996448]  0000000000004000 0000000000000000 ffff8801375a2000
> 0000000000000000
> [ 2743.998918] Call Trace:
> [ 2744.000205]  [<ffffffff8115b7b7>] oom_kill_process+0x47/0x50
> [ 2744.002173]  [<ffffffff811be85c>] mem_cgroup_out_of_memory+0x28c/0x2b0
> [ 2744.004317]  [<ffffffff811c0c0b>] mem_cgroup_oom_synchronize+0x21b/0x250
> [ 2744.006492]  [<ffffffff811c04c0>] ? memcg_charge_kmem+0xf0/0xf0
> [ 2744.008507]  [<ffffffff8115bb98>] pagefault_out_of_memory+0x18/0x90
> [ 2744.010597]  [<ffffffff817453c4>] mm_fault_error+0xb9/0xd3
> [ 2744.012529]  [<ffffffff817638d5>] __do_page_fault+0x545/0x570
> [ 2744.014536]  [<ffffffff811182cc>] ? acct_account_cputime+0x1c/0x20
> [ 2744.016601]  [<ffffffff810a25e9>] ? account_user_time+0x99/0xb0
> [ 2744.018597]  [<ffffffff810a2c6d>] ? vtime_account_user+0x5d/0x70
> [ 2744.020595]  [<ffffffff8176391a>] do_page_fault+0x1a/0x70
> [ 2744.193452]  [<ffffffff8175fa88>] page_fault+0x28/0x30
> [ 2744.196140] Code: c7 08 50 a7 81 31 c0 e8 1c e1 ff ff 4c 89 e7 e8
> b1 66 01 00 48 c7 c7 40 60 c0 81 e8 05 63 01 00 48 89 5d a8 4d 8b a7
> 08 03 00 00 <4d> 8d af 08 03 00 00 49 81 ec 18 03 00 00 eb 57 48 8b 83
> a8 02
> [ 2771.854441] BUG: soft lockup - CPU#0 stuck for 22s! [php5:15940]
> 
> 
> 3. lxc, cgroup, OOM, linux-image-3.13.0-32-generic,
> linux-image-3.13.0-35-generic, vsphere
> 
> 
> 4. Linux version 3.13.0-35-generic (buildd@akateko) (gcc version 4.6.3
> (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #62~precise1-Ubuntu SMP Mon Aug 18
> 14:52:04 UTC 2014
> 
> 
> 7.1
> 
> stackato@stackato-h5jt:~$ ./ver_linux.sh
> ./ver_linux.sh: line 1: ng: command not found
> If some fields are empty or look unusual you may have an old version.
> Compare to the current minimal requirements in Documentation/Changes.
> 
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=12.04
> DISTRIB_CODENAME=precise
> DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"
> NAME="Ubuntu"
> VERSION="12.04.4 LTS, Precise Pangolin"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu precise (12.04.4 LTS)"
> VERSION_ID="12.04"
> Linux stackato-h5jt 3.13.0-35-generic #62~precise1-Ubuntu SMP Mon Aug
> 18 14:52:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> 
> Gnu C                  gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
> Gnu make               3.81
> util-linux             linux 2.20.1
> mount                  linux 2.20.1 (with libblkid and selinux support)
> modutils               3.16
> e2fsprogs              1.42
> PPP                    2.4.5
> Linux C Library        > libc.2.15
> Dynamic linker (ldd)   2.15
> Procps                 3.2.8
> Net-tools              1.60
> iproute2              iproute2-ss111117
> Kbd                    1.15.2
> Sh-utils               8.13
> Modules Loaded         xt_nat veth ipt_REJECT xt_tcpudp xt_state
> xt_addrtype xt_conntrack iptable_filter aufs ipt_MASQUERADE
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack ip_tables x_tables bridge stp llc vmwgfx ttm psmouse
> vmw_vmci drm vmw_balloon shpchp mac_hid i2c_piix4 serio_raw ppdev
> parport_pc lp parport floppy mptspi mptscsih e1000 mptbase vmw_pvscsi
> vmxnet3
> 
> free reports:
>              total       used       free     shared    buffers     cached
> Mem:       1918540     786944    1131596          0      82684     278140
> -/+ buffers/cache:     426120    1492420
> Swap:      1951740          0    1951740
> 
> 
> 7.2
> 
> /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 44
> model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
> stepping : 2
> microcode : 0x13
> cpu MHz : 2400.085
> cache size : 12288 KB
> physical id : 0
> siblings : 1
> core id : 0
> cpu cores : 1
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable
> nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor
> lahf_lm ida arat epb dtherm
> bogomips : 4800.17
> clflush size : 64
> cache_alignment : 64
> address sizes : 40 bits physical, 48 bits virtual
> power management:
> 
> processor : 1
> vendor_id : GenuineIntel
> cpu family : 6
> model : 44
> model name : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
> stepping : 2
> microcode : 0x13
> cpu MHz : 2400.085
> cache size : 12288 KB
> physical id : 2
> siblings : 1
> core id : 0
> cpu cores : 1
> apicid : 2
> initial apicid : 2
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable
> nonstop_tsc aperfmperf pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor
> lahf_lm ida arat epb dtherm
> bogomips : 4800.17
> clflush size : 64
> cache_alignment : 64
> address sizes : 40 bits physical, 48 bits virtual
> power management:
> 
> 7.3
> 
> cat /proc/modules
> xt_nat 12726 26 - Live 0x0000000000000000
> veth 13331 0 - Live 0x0000000000000000
> ipt_REJECT 12576 0 - Live 0x0000000000000000
> xt_tcpudp 12924 39 - Live 0x0000000000000000
> xt_state 12578 13 - Live 0x0000000000000000
> xt_addrtype 12713 2 - Live 0x0000000000000000
> xt_conntrack 12760 1 - Live 0x0000000000000000
> iptable_filter 12810 1 - Live 0x0000000000000000
> aufs 210175 999 - Live 0x0000000000000000
> ipt_MASQUERADE 12880 2 - Live 0x0000000000000000
> iptable_nat 13151 1 - Live 0x0000000000000000
> nf_conntrack_ipv4 15063 15 - Live 0x0000000000000000
> nf_defrag_ipv4 12758 1 nf_conntrack_ipv4, Live 0x0000000000000000
> nf_nat_ipv4 13316 1 iptable_nat, Live 0x0000000000000000
> nf_nat 26091 4 xt_nat,ipt_MASQUERADE,iptable_nat,nf_nat_ipv4, Live
> 0x0000000000000000
> nf_conntrack 97581 7
> xt_state,xt_conntrack,ipt_MASQUERADE,iptable_nat,nf_conntrack_ipv4,nf_nat_ipv4,nf_nat,
> Live 0x0000000000000000
> ip_tables 27716 2 iptable_filter,iptable_nat, Live 0x0000000000000000
> x_tables 34194 9
> xt_nat,ipt_REJECT,xt_tcpudp,xt_state,xt_addrtype,xt_conntrack,iptable_filter,ipt_MASQUERADE,ip_tables,
> Live 0x0000000000000000
> bridge 116291 0 - Live 0x0000000000000000
> stp 12976 1 bridge, Live 0x0000000000000000
> llc 14597 2 bridge,stp, Live 0x0000000000000000
> vmwgfx 185498 1 - Live 0x0000000000000000
> ttm 90134 1 vmwgfx, Live 0x0000000000000000
> psmouse 113295 0 - Live 0x0000000000000000
> vmw_vmci 68525 0 - Live 0x0000000000000000
> drm 308868 2 vmwgfx,ttm, Live 0x0000000000000000
> vmw_balloon 13593 0 - Live 0x0000000000000000
> shpchp 37201 0 - Live 0x0000000000000000
> mac_hid 13253 0 - Live 0x0000000000000000
> i2c_piix4 22299 0 - Live 0x0000000000000000
> serio_raw 13462 0 - Live 0x0000000000000000
> ppdev 17711 0 - Live 0x0000000000000000
> parport_pc 32866 1 - Live 0x0000000000000000
> lp 17799 0 - Live 0x0000000000000000
> parport 42481 3 ppdev,parport_pc,lp, Live 0x0000000000000000
> floppy 70207 0 - Live 0x0000000000000000
> mptspi 22921 3 - Live 0x0000000000000000
> mptscsih 44751 1 mptspi, Live 0x0000000000000000
> e1000 152011 0 - Live 0x0000000000000000
> mptbase 103162 2 mptspi,mptscsih, Live 0x0000000000000000
> vmw_pvscsi 23372 0 - Live 0x0000000000000000
> vmxnet3 50657 0 - Live 0x0000000000000000
> 
> Thanks,
> Tyler

-- 
tejun