public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* rsync out of memory 2.6.8-rc2
@ 2004-07-18 21:52 Klaus Dittrich
  2004-07-19  7:48 ` Nick Piggin
  0 siblings, 1 reply; 8+ messages in thread
From: Klaus Dittrich @ 2004-07-18 21:52 UTC (permalink / raw)
  To: linux mailing-list

rsync-2.6.2 of a large disc using 2.6.8-rc2 (I also tried 2.6.7-bk14)
stops with "kernel: Out of Memory: Killed process xxxx" messages
during filelist gathering.

When this happens only 1.4 GB out of 2 GB RAM and no swap is used.

No problems with kernel 2.6.7.
(Peak RAM usage during filelist gathering was 1.8 GB, no swap)

--
Klaus


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: rsync out of memory 2.6.8-rc2
  2004-07-18 21:52 Klaus Dittrich
@ 2004-07-19  7:48 ` Nick Piggin
  2004-07-19  7:52   ` bert hubert
  0 siblings, 1 reply; 8+ messages in thread
From: Nick Piggin @ 2004-07-19  7:48 UTC (permalink / raw)
  To: Klaus Dittrich; +Cc: linux mailing-list

Klaus Dittrich wrote:
> rsync-2.6.2 of a large disc using 2.6.8-rc2 (I also tried 2.6.7-bk14)
> stops with "kernel: Out of Memory: Killed process xxxx" messages
> during filelist gathering.
> 
> When this happens only 1.4 GB out of 2 GB RAM and no swap is used.
> 
> No problems with kernel 2.6.7.
> (Peak RAM usage during filelist gathering was 1.8 GB, no swap)
> 

Hi Klaus,
What arguments are you running rsync with? Also, how many files, and
how large are they?

Thanks
Nick

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: rsync out of memory 2.6.8-rc2
  2004-07-19  7:48 ` Nick Piggin
@ 2004-07-19  7:52   ` bert hubert
  2004-07-19  8:47     ` Klaus Dittrich
  0 siblings, 1 reply; 8+ messages in thread
From: bert hubert @ 2004-07-19  7:52 UTC (permalink / raw)
  To: Nick Piggin; +Cc: Klaus Dittrich, linux mailing-list

On Mon, Jul 19, 2004 at 05:48:25PM +1000, Nick Piggin wrote:
> Klaus Dittrich wrote:
> >rsync-2.6.2 of a large disc using 2.6.8-rc2 (I also tried 2.6.7-bk14)
> >stops with "kernel: Out of Memory: Killed process xxxx" messages
> >during filelist gathering.
> >
> >When this happens only 1.4 GB out of 2 GB RAM and no swap is used.
> >
> >No problems with kernel 2.6.7.
> >(Peak RAM usage during filelist gathering was 1.8 GB, no swap)
> >
> 
> Hi Klaus,
> What arguments are you running rsync with? Also, how many files, and
> how large are they?

And can you show the output of 'vmstat 1' during rsync? This is the second
report of linux 2.6.8-rc? eating all memory, btw.

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: rsync out of memory 2.6.8-rc2
  2004-07-19  7:52   ` bert hubert
@ 2004-07-19  8:47     ` Klaus Dittrich
  0 siblings, 0 replies; 8+ messages in thread
From: Klaus Dittrich @ 2004-07-19  8:47 UTC (permalink / raw)
  To: bert hubert; +Cc: Nick Piggin, Klaus Dittrich, linux mailing-list

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

bert hubert wrote:

>On Mon, Jul 19, 2004 at 05:48:25PM +1000, Nick Piggin wrote:
>  
>
>>Klaus Dittrich wrote:
>>    
>>
>>>rsync-2.6.2 of a large disc using 2.6.8-rc2 (I also tried 2.6.7-bk14)
>>>stops with "kernel: Out of Memory: Killed process xxxx" messages
>>>during filelist gathering.
>>>
>>>When this happens only 1.4 GB out of 2 GB RAM and no swap is used.
>>>
>>>No problems with kernel 2.6.7.
>>>(Peak RAM usage during filelist gathering was 1.8 GB, no swap)
>>>
>>>      
>>>
>>Hi Klaus,
>>What arguments are you running rsync with? Also, how many files, and
>>how large are they?
>>    
>>
>
>And can you show the output of 'vmstat 1' during rsync? This is the second
>report of linux 2.6.8-rc? eating all memory, btw.
>
>  
>
Fine to know that it is no local problem. Output of vmstat 1 is appended.
--
Klaus

[-- Attachment #2: vmmstat_during_rsync --]
[-- Type: text/plain, Size: 61383 bytes --]

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 1826920   6016 126856    0    0    69    16  268   147  1  0 96  2
 0  0      0 1826920   6016 126856    0    0     0     0 1179   837  1  0 99  0
 0  1      0 1826408   6120 127160    0    0   424     0 1188   684  1  0 91  8
 0  1      0 1824360   6532 127428    0    0   708     0 1243   700  0  1 75 24
 1  1      0 1823080   6836 127668    0    0   568     0 1193   601  0  1 75 25
 0  1      0 1821480   7184 128000    0    0   624     0 1212   695  0  0 75 24
 0  1      0 1819752   7544 128320    0    0   644     0 1216   692  0  1 75 25
 0  1      0 1818344   7892 128584    0    0   644     0 1206   708  0  0 75 24
 1  1      0 1817384   8184 128904    0    0   572     0 1192   690  0  0 75 25
 0  1      0 1814744   8432 129268    0    0   644     0 1293   799  1  1 74 24
 0  1      0 1811528   8736 129712    0    0   760     0 1563  1401  1  2 75 23
 1  1      0 1810184   8804 130052    0    0   428    72 1387   974  1  1 75 24
 0  1      0 1808136   8968 130364    0    0   512     0 1327  2057  2  1 73 24
 0  1      0 1806792   9044 130696    0    0   412     0 1166   747  0  1 75 24
 0  1      0 1805064   9200 131084    0    0   508     0 1192   680  0  0 75 24
 2  2      0 1803080   9392 131572    0    0   532     0 1210  1008  1  1 74 24
 0  1      0 1801800   9500 131872    0    0   460   100 1204   934  1  0 74 25
 0  1      0 1800072   9696 132152    0    0   448     0 1179   785  0  1 75 24
 0  1      0 1798088   9920 132404    0    0   488     0 1179   633  1  1 75 24
 0  1      0 1796360  10104 132696    0    0   448     0 1178   708  0  1 75 24
 0  1      0 1795144  10240 132968    0    0   408     0 1166  1146  1  0 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1793800  10368 133248    0    0   428     0 1169   684  0  1 74 24
 0  1      0 1792520  10484 133540    0    0   428     0 1169   674  0  0 74 25
 0  1      0 1790216  10780 133856    0    0   576     0 1211   670  1  1 74 25
 0  1      0 1788248  10960 134152    0    0   448     0 1172   562  1  0 74 24
 0  1      0 1786072  11188 135216    0    0  1296     0 1391  1000  1  1 75 24
 0  1      0 1784440  11388 135492    0    0   464     0 1175   574  1  0 75 25
 0  1      0 1780024  11992 135704    0    0   824     0 1307   787  1  2 74 23
 0  1      0 1778296  12248 135856    0    0   468     0 1175   593  0  1 75 24
 0  1      0 1777080  12360 136152    0    0   368     0 1153   571  1  1 75 25
 0  1      0 1776184  12408 136444    0    0   372     0 1140   503  0  0 74 25
 0  1      0 1774968  12512 136748    0    0   400    12 1159   541  0  0 75 24
 0  1      0 1773176  12708 137028    0    0   488     0 1179   590  1  1 75 24
 1  0      0 1768888  13276 137344    0    0   804     0 1302   843  1  2 74 23
 0  1      0 1766840  13476 137552    0    0   448     0 1174   607  1  1 74 24
 0  1      0 1765432  13604 137832    0    0   404     0 1165   591  1  1 74 24
 0  1      0 1764472  13624 138220    0    0   384     4 1145   572  1  0 74 25
 0  1      0 1761848  13784 139352    0    0  1284     0 1399  1055  1  1 74 24
 0  1      0 1760184  13976 139568    0    0   420     0 1164   615  1  1 74 24
 0  1      0 1756728  14412 139812    0    0   684     0 1252   768  1  1 74 23
 0  1      0 1754008  14792 140044    0    0   580     0 1221   704  1  1 74 24
 0  1      0 1753048  14904 140272    0    0   352     0 1137   574  1  0 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1751832  15032 140552    0    0   396     0 1152   598  1  1 74 24
 0  1      0 1750808  15072 140852    0    0   348     0 1140   592  1  0 74 25
 0  1      0 1749016  15268 141132    0    0   480     0 1406   975  1  1 74 24
 0  1      0 1747480  15444 141364    0    0   420     0 1325  1809  2  1 73 24
 1  0      0 1742592  16040 141584    0    0   812   100 1505  2754 15  2 61 21
 0  1      0 1740928  16288 141812    0    0   472     0 1417  2351  8  2 68 23
 0  1      0 1739456  16432 142076    0    0   408     0 1170   585  1  0 74 24
 0  1      0 1738752  16472 142376    0    0   356     0 1131   579  1  0 75 25
 0  1      0 1736912  16660 142664    0    0   468     0 1181   658  1  1 75 24
 0  1      0 1735376  16824 142908    0    0   416     4 1337  1228  1  1 74 24
 0  2      0 1727552  17128 146752    0    0  4224     0 1485  2173  4  2 70 24
 0  2      0 1714296  17356 156724    0    0  9936     0 1449  1443  4  1 70 24
 3  1      0 1703224  17572 161676    0    0  5096     0 1322  2173 15  2 59 24
 0  1      0 1696560  17796 164512    0    0  2832    40 1396  3580 14  2 58 26
 0  1      0 1695536  17904 164744    0    0   360   248 1431  1634  5  1 71 23
 0  1      0 1694576  18060 164996    0    0   404     0 1179   850  1  0 75 24
 0  1      0 1693616  18156 165240    0    0   344     0 1155   775  1  0 75 24
 0  1      0 1692080  18264 165608    0    0   412     0 1185   887  1  0 74 24
 0  1      0 1691056  18364 165848    0    0   364     0 1157   736  0  0 75 24
 1  1      0 1690096  18440 166112    0    0   344     0 1153   815  1  0 75 25
 0  1      0 1687728  18608 166420    0    0   408     0 1176  1946  8  1 68 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1685952  18840 166596    0    0   452     8 1196   827  0  1 75 24
 0  1      0 1683968  19076 166836    0    0   440     0 1192   784  1  1 75 24
 0  1      0 1682752  19208 167044    0    0   400     0 1166   757  1  0 75 24
 2  2      0 1681600  19328 167332    0    0   352  1028 1352   865  1  1 74 24
 0  2      0 1680256  19428 167572    0    0   368   304 1298  2387  6  1 61 33
 0  2      0 1678720  19628 167780    0    0   424   400 1338  1027  0  1 74 25
 0  1      0 1677056  19776 168040    0    0   388   256 1515  2336  9  1 66 23
 0  1      0 1675904  19900 168324    0    0   416     0 1349  2901  8  1 67 23
 0  1      0 1675200  19980 168516    0    0   256     8 1167   722  1  0 75 24
 0  1      0 1674240  20092 168676    0    0   280     0 1143   881  2  1 74 24
 1  1      0 1671456  20320 169740    0    0  1228  2632 1900  3385  8  2 66 24
 3  1      0 1669856  20552 169916    0    0   452     0 1528  1963  8  2 69 22
 0  1      0 1667296  20808 170136    0    0   492     0 1377  2171  9  1 67 23
 0  1      0 1666208  20964 170388    0    0   404     8 1177  1719  6  1 69 24
 0  1      0 1664992  21096 170664    0    0   372     0 1357  1537  1  1 74 24
 0  1      0 1663776  21176 170924    0    0   344     0 1402  2173  6  1 69 23
 0  1      0 1662568  21320 171120    0    0   392     0 1268   907  1  0 75 24
 0  1      0 1660648  21516 171400    0    0   432     0 1408  1195  2  1 74 24
 1  1      0 1658984  21756 171636    0    0   496    16 1271   975  1  1 74 24
 0  1      0 1657320  21872 172064    0    0   516     0 1180  1349  9  1 67 23
 0  1      0 1656120  22012 172332    0    0   384     0 1165   628  0  0 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1654520  22232 172588    0    0   532     0 1203   709  1  1 74 24
 0  1      0 1651896  22512 172920    0    0   600     0 1441  2000  5  2 70 23
 0  1      0 1649768  22744 173164    0    0   536   100 1463  2001 10  1 66 23
 0  1      0 1648104  22924 173460    0    0   432     0 1306   880  1  1 74 24
 1  0      0 1646952  23028 173696    0    0   400     0 1242   775  1  1 74 24
 0  1      0 1644792  23256 173944    0    0   460     0 1261   791  1  1 75 24
 0  1      0 1642872  23452 174360    0    0   624     0 1284   820  1  0 75 24
 0  1      0 1640064  23808 174684    0    0   644    92 1314   895  1  1 75 24
 0  1      0 1639040  23884 174948    0    0   352     0 1283   950  1  0 74 24
 0  1      0 1638624  23908 175196    0    0   316     0 1149   635  0  0 75 24
 0  1      0 1638112  23960 175484    0    0   324     0 1158   636  0  0 75 24
 0  1      0 1637616  23992 175792    0    0   324     0 1133   593  0  0 75 25
 0  1      0 1637040  24024 176100    0    0   340     0 1137   594  0  0 75 25
 0  1      0 1636656  24056 176408    0    0   340     0 1136   624  0  0 75 25
 1  1      0 1636080  24076 176728    0    0   360     0 1213   672  1  0 74 24
 0  1      0 1635440  24120 177024    0    0   328     0 1493  2930  7  1 69 23
 0  1      0 1634544  24184 177368    0    0   400     0 1286  1009  1  1 74 24
 0  1      0 1633968  24200 177760    0    0   424    12 1162   794  1  0 75 24
 0  1      0 1632304  24392 178044    0    0   416  1108 1492   951  1  1 74 24
 0  1      0 1631024  24524 178252    0    0   348     8 1267   772  1  1 75 24
 0  1      0 1630640  24528 178452    0    0   248     0 1114   612  1  0 74 25
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1630080  24580 178672    0    0   276     0 1124   560  1  0 74 24
 0  1      0 1629312  24608 179052    0    0   364   456 1226   666  1  0 74 25
 0  1      0 1628736  24652 179348    0    0   368     0 1143   680  0  0 75 25
 0  1      0 1627792  24744 179596    0    0   376     0 1153   657  1  0 75 25
 2  1      0 1625680  24984 179968    0    0   580     0 1217   784  1  1 75 24
 0  1      0 1624720  25060 180368    0    0   476     0 1180   728  1  0 75 24
 0  1      0 1622672  25288 180548    0    0   440     4 1184   738  1  1 74 24
 0  1      0 1620624  25532 180780    0    0   500     0 1201   772  1  1 74 24
 0  1      0 1619344  25648 181140    0    0   476     0 1181   733  1  0 75 25
 0  1      0 1618576  25708 181420    0    0   340     0 1139   667  1  0 74 25
 0  1      0 1616528  25928 181880    0    0   584     0 1227   805  1  1 75 24
 0  1      0 1615376  26056 182160    0    0   444     4 1174   724  1  1 74 24
 0  1      0 1614480  26136 182420    0    0   352     0 1142   626  1  0 74 25
 0  1      0 1613008  26300 182664    0    0   400     0 1168   624  1  0 74 24
 0  1      0 1611472  26464 182908    0    0   396     0 1167   636  1  0 75 24
 0  1      0 1610448  26588 183192    0    0   372     0 1154   665  1  1 74 25
 0  1      0 1609488  26680 183440    0    0   400     0 1155   699  0  0 74 25
 0  1      0 1607824  26840 183756    0    0   440     0 1179   750  1  0 74 24
 0  1      0 1606608  26972 184100    0    0   464     0 1179   721  1  0 75 24
 0  1      0 1605648  27056 184424    0    0   432     0 1167   709  1  0 74 25
 0  1      0 1605072  27092 184728    0    0   348     0 1138   659  1  0 75 25
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1604240  27184 185044    0    0   384     0 1151   688  1  0 74 24
 0  1      0 1603216  27276 185292    0    0   368     0 1154   704  1  0 74 25
 0  1      0 1602128  27384 185524    0    0   368     0 1157   685  1  1 75 25
 0  1      0 1601040  27480 185836    0    0   348     0 1145   603  1  0 75 24
 0  1      0 1599952  27608 186048    0    0   364     0 1156   607  1  0 74 24
 0  1      0 1598672  27740 186324    0    0   380     4 1157   628  1  0 75 25
 0  1      0 1597584  27840 186632    0    0   348     0 1148   625  1  0 74 25
 0  1      0 1596432  27936 186876    0    0   376     0 1155   614  1  0 75 25
 0  1      0 1594064  28312 187248    0    0   776     0 1270   831  1  1 75 24
 0  1      0 1592464  28604 187568    0    0   560     0 1206   773  1  0 74 24
 0  1      0 1589904  29044 188012    0    0   872    36 1306   979  1  1 74 24
 0  1      0 1588432  29452 188284    0    0   748     0 1247   897  1  1 74 24
 0  1      0 1586576  29868 188548    0    0   624     0 1231   815  1  0 74 24
 0  1      0 1583696  30392 188772    0    0   784     0 1273   895  1  1 75 24
 0  1      0 1582288  30764 189080    0    0   700     0 1244   854  1  1 74 24
 0  1      0 1578640  31448 189348    0    0   928   916 1567  1044  1  2 74 23
 0  1      0 1575760  31984 189492    0    0   720     0 1259   816  1  1 74 23
 0  1      0 1570192  32896 189804    0    0  1168     0 1417  1050  1  2 75 22
 0  1      0 1565968  33592 190060    0    0   936     0 1343   940  1  1 74 23
 0  1      0 1563152  34120 190552    0    0  1032     0 1345   970  1  1 74 24
 0  1      0 1559632  34560 191608    0    0  1512    72 1476  1221  1  1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1556368  34920 193152    0    0  1888     0 1542  1409  1  1 74 24
 0  1      0 1552288  35496 193596    0    0  1000     0 1338   956  1  2 74 23
 1  0      0 1548576  36004 193972    0    0   904     0 1319   960  1  1 74 24
 0  1      0 1544800  36500 194700    0    0  1248     0 1408  1146  1  1 75 24
 0  1      0 1542560  36936 195012    0    0   764     0 1263   897  1  1 75 24
 0  1      0 1541024  37292 195200    0    0   596     0 1216   821  1  0 74 24
 0  1      0 1539040  37736 195572    0    0   736     0 1256   787  1  1 74 24
 1  0      0 1536608  38332 195860    0    0   892     0 1305   874  1  1 75 24
 0  1      0 1534304  38704 196304    0    0   784     0 1271   827  1  1 74 24
 0  1      0 1532832  39180 196644    0    0   860     0 1276   865  1  1 75 24
 0  1      0 1531872  39508 196928    0    0   592     0 1204   744  1  0 74 25
 0  1      0 1528800  40128 197192    0    0   872     0 1301   881  1  1 74 24
 0  1      0 1525472  40876 197464    0    0  1048     0 1401  1082  1  1 74 23
 1  1      0 1523552  41280 197672    0    0   620     0 1497  2305  4  2 72 23
 1  0      0 1520864  41928 198044    0    0  1032     0 1678  4455 17  3 60 20
 0  1      0 1518112  42444 198276    0    0   776     0 1527  1801  2  2 74 23
 0  1      0 1516704  42716 198616    0    0   556     0 1474  1726  1  1 74 24
 0  1      0 1514848  43044 198968    0    0   652     0 1290   939  1  1 74 24
 0  1      0 1513440  43344 199144    0    0   552     0 1295   908  1  0 74 24
 0  1      0 1510880  43796 199508    0    0   768     0 1269   973  0  1 75 24
 0  1      0 1508384  44284 199768    0    0   696     0 1253   902  1  1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1505952  44632 200100    0    0   708     0 1258  1250  1  1 75 24
 0  1      0 1504160  44944 200468    0    0   708     0 1254  1355  1  1 75 24
 0  1      0 1502944  45156 200664    0    0   416     0 1171   817  0  0 75 24
 0  1      0 1500000  45632 201140    0    0   836     4 1308  1014  0  1 75 24
 0  1      0 1495072  46184 201676    0    0  1136     0 1403  1147  1  2 75 23
 0  1      0 1493536  46356 202048    0    0   516     0 1202  1169  1  0 75 24
 0  1      0 1492704  46488 202324    0    0   448     0 1171   834  1  0 74 24
 0  1      0 1490848  46928 202768    0    0   868     0 1285  1026  0  1 75 24
 0  1      0 1486944  47364 204032    0    0  1684  1424 1763  2059  1  2 72 26
 0  1      0 1485984  47556 204248    0    0   448     8 1328   835  1  1 74 24
 0  1      0 1485280  47756 204388    0    0   380     0 1151   650  0  0 75 25
 0  1      0 1484256  47964 204656    0    0   496     0 1180   714  0  0 75 25
 0  1      0 1482016  48356 205284    0    0   960     0 1382  1096  0  1 75 24
 0  1      0 1479968  48692 205968    0    0  1028    20 1331  1084  0  0 75 24
 0  1      0 1477152  49096 206720    0    0  1224     0 1381  1164  1  1 74 24
 0  1      0 1475232  49380 207388    0    0   952     0 1306   970  1  1 74 24
 0  1      0 1473184  49648 208208    0    0  1092     0 1348  1050  1  1 74 24
 0  1      0 1469536  50004 209552    0    0  1684     0 1504  1368  1  1 75 24
 0  1      0 1466208  50328 210792    0    0  1632     0 1482  1335  1  1 74 23
 0  1      0 1464224  50604 211468    0    0   876     0 1285   966  1  0 74 24
 0  1      0 1462240  50872 212152    0    0   976     0 1314   943  1  1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1459616  51184 212928    0    0  1100     0 1354  1002  1  1 75 24
 0  1      0 1456992  51504 213424    0    0   812     0 1287   847  1  1 74 24
 1  1      0 1454752  51816 214132    0    0   984     4 1320   939  1  1 74 24
 0  1      0 1452320  52076 214960    0    0  1124     0 1354  1020  1  1 74 24
 0  1      0 1450336  52332 215520    0    0   748     0 1258   874  1  1 74 24
 0  1      0 1446560  52664 216684    0    0  1580     0 1480  1290  1  1 75 23
 0  1      0 1443296  52984 217588    0    0  1164     0 1380  1079  1  1 74 24
 0  1      0 1438496  53504 218836    0    0  1800     0 1554  1434  1  2 74 23
 0  1      0 1434144  53892 220148    0    0  1700     0 1514  1382  1  1 75 23
 1  0      0 1431712  54220 221112    0    0  1288     0 1393  1143  1  1 74 24
 0  1      0 1428896  54552 221936    0    0  1216     0 1381  1102  1  1 74 24
 0  1      0 1426272  54960 222956    0    0  1344     0 1409  1105  1  1 74 24
 0  1      0 1423264  55252 224024    0    0  1380     0 1436  1130  1  1 75 23
 0  1      0 1420960  55352 225284    0    0  1328     0 1433  1135  1  1 74 24
 0  1      0 1419168  55440 226284    0    0  1156     0 1374  1010  1  1 74 24
 0  1      0 1417696  55596 226876    0    0   756     0 1272   820  1  1 74 25
 0  1      0 1414752  55992 227432    0    0   908     0 1342   924  1  1 74 24
 0  1      0 1412960  56408 227696    0    0   728     8 1261   817  1  0 74 24
 0  1      0 1410912  56780 228004    0    0   684     0 1245   829  1  1 75 24
 0  1      0 1408928  57100 228500    0    0   748     0 1259   880  1  1 74 24
 0  1      0 1407072  57380 228900    0    0   756     0 1256   885  1  1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1405472  57684 229276    0    0   588     0 1214   792  1  0 74 24
 0  1      0 1404256  57984 229520    0    0   572    24 1207   846  1  0 75 25
 0  1      0 1402592  58272 229912    0    0   652     0 1236   781  1  1 74 24
 0  1      0 1400288  58664 230268    0    0   716     0 1254   783  1  1 75 24
 0  1      0 1398944  59000 230544    0    0   556     0 1209   705  1  0 74 25
 0  1      0 1396192  59560 230800    0    0   908     0 1312   892  1  1 74 24
 0  2      0 1393888  59952 230952    0    0   532  3436 1658   850  1  1 65 33
 0  1      0 1393440  60032 231076    0    0   156   216 1437   619  0  1 71 28
 0  1      0 1391456  60512 231480    0    0   896     8 1303   931  1  0 74 24
 0  1      0 1390112  60872 231732    0    0   656     0 1226   774  1  1 74 25
 0  1      0 1386464  61364 232396    0    0  1124     0 1370  1018  1  1 74 24
 0  1      0 1382944  61812 233308    0    0  1380    20 1431  1143  1  1 74 23
 0  1      0 1380064  62244 233964    0    0  1060     0 1342  1045  1  1 74 24
 0  1      0 1377568  62660 234568    0    0  1024     0 1333  1028  1  1 74 24
 0  1      0 1373216  63244 235412    0    0  1380     0 1467  1160  1  1 74 23
 0  1      0 1370016  63624 236460    0    0  1516     0 1498  1219  1  1 74 24
 0  1      0 1366368  63832 238360    0    0  2056    64 1658  1492  1  2 74 23
 1  0      0 1356640  65000 239912    0    0  2708     0 1972  1950  2  3 73 21
 0  1      0 1352928  65508 241036    0    0  1660     0 1541  1311  1  1 74 24
 0  1      0 1349024  66020 241952    0    0  1444     0 1489  1192  1  1 74 23
 0  1      0 1347488  66220 242500    0    0   760     0 1265   816  1  1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1345888  66472 242996    0    0   752     0 1271   840  1  1 74 24
 0  1      0 1342432  66704 244464    0    0  1720     0 1537  1362  1  1 74 24
 0  1      0 1337696  66876 247760    0    0  3472     0 1972  2211  1  2 74 23
 0  1      0 1333152  67064 251040    0    0  3444     0 1963  2179  1  1 75 23
 0  1      0 1330272  67172 253040    0    0  2136     0 1605  1494  1  1 74 24
 0  1      0 1325920  67400 255260    0    0  2456     0 1727  1693  1  2 74 23
 0  1      0 1321632  67656 257384    0    0  2416     0 1729  1739  1  2 74 23
 0  1      0 1318880  67944 258116    0    0   956     0 1348   968  1  1 74 24
 0  1      0 1317216  68228 258716    0    0   864     0 1276   899  1  1 75 24
 0  1      0 1313888  68624 259340    0    0  1052     0 1339  1017  1  1 74 24
 1  1      0 1310880  68992 260196    0    0  1256     4 1382  1105  1  1 75 24
 0  1      0 1308832  69316 260552    0    0   676     0 1234   837  1  0 74 24
 0  1      0 1306336  69816 261004    0    0   964     0 1304   964  1  1 74 24
 0  1      0 1304608  70068 261500    0    0   712     0 1237   839  1  0 75 24
 0  1      0 1303136  70380 261800    0    0   600     0 1200   766  1  0 74 24
 0  1      0 1301344  70640 262084    0    0   540     0 1194   762  1  0 74 24
 0  1      0 1299232  70916 262556    0    0   772     0 1258   901  1  1 75 24
 0  1      0 1297504  71296 262788    0    0   632     0 1362   950  1  1 74 24
 1  1      0 1295904  71612 263356    0    0   848     0 1602  1978  1  1 73 24
 0  1      0 1293664  71980 263736    0    0   736     0 1312   965  1  1 74 24
 0  1      0 1291344  72340 264124    0    0   748    44 1267  1242  5  2 69 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1289352  72792 264556    0    0   912     0 1546  1745  1  1 74 24
 0  1      0 1286024  73300 265204    0    0  1136     0 1487  2588  3  2 73 23
 1  0      0 1282120  73752 266248    0    0  1508     0 1534  1417  1  1 75 23
 0  1      0 1278024  74184 267244    0    0  1396     0 1434  1381  1  2 74 23
 0  2      0 1275912  74452 267588    0    0   660  4072 1707  1112  1  2 66 31
 0  1      0 1274952  74540 267772    0    0   304    40 1377   951  1  1 74 25
 0  1      0 1270936  74908 268220    0    0   784     0 1287  1088  2  2 72 24
 0  1      0 1268824  75252 268760    0    0   884     0 1315   967  1  1 74 24
 0  1      0 1265192  75408 271324    0    0  2672     0 1774  2402  8  2 66 23
 1  1      0 1261928  75644 273264    0    0  2156     0 1653  1754  4  2 71 23
 0  1      0 1256040  76644 274372    0    0  2128     0 1700  1583  1  2 74 23
 0  1      0 1250856  77028 274872    0    0   892     0 1407  1389  6  3 69 23
 0  1      0 1249256  77132 275924    0    0  1152     0 1350  1311  4  1 71 24
 0  1      0 1246512  77468 276812    0    0  1216     0 1405  1051  1  1 74 24
 0  1      0 1244080  77756 277272    0    0   744    12 1272   892  0  1 74 24
 0  1      0 1242672  77932 277708    0    0   648     0 1227   833  1  0 74 24
 0  1      0 1239920  78200 278732    0    0  1256     0 1412  1135  1  1 74 24
 0  1      0 1236672  78316 280928    0    0  2352     0 1663  1678  1  1 75 24
 0  1      0 1233088  78404 283696    0    0  2848     0 1866  2037  1  1 74 24
 0  2      0 1230080  78504 285908    0    0  2312  2256 2222  3102  1  2 67 31
 1  1      0 1228800  78532 286832    0    0   976  1068 1728  1891  1  1 64 34
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1227712  78548 287700    0    0   920     8 1444  1220  0  1 74 24
 0  1      0 1224832  78632 289656    0    0  2008     0 1577  1430  1  1 75 24
 1  1      0 1221312  78864 291056    0    0  1684     0 1539  1333  1  1 74 23
 0  1      0 1217696  79056 292496    0    0  1608    36 1539  1487  2  1 73 23
 0  1      0 1214944  79324 293248    0    0   976     0 1329   943  0  1 74 24
 0  1      0 1212192  79824 293904    0    0  1176     0 1353  1021  1  1 75 24
 0  1      0 1207648  80332 294824    0    0  1452     0 1448  1171  1  1 74 23
 0  1      0 1203872  80744 295772    0    0  1368     0 1416  1185  1  1 74 23
 0  1      0 1200216  81120 296620    0    0  1220    12 1389  1120  1  2 75 23
 0  1      0 1195672  81668 297432    0    0  1388     0 1482  1267  1  2 75 23
 0  1      0 1191896  81812 299940    0    0  2656     0 1763  1841  1  1 74 24
 0  1      0 1188312  82028 302036    0    0  2224     0 1663  1563  1  1 74 24
 0  1      0 1178904  83196 303384    0    0  2392     0 1870  1755  1  4 74 21
 1  1      0 1176472  83416 304388    0    0  1220     4 1395  1055  1  1 74 24
 0  1      0 1173272  83704 305664    0    0  1572     0 1496  1247  1  1 74 24
 0  1      0 1170584  84004 306180    0    0   864     0 1307   858  1  1 75 24
 0  1      0 1168472  84200 306868    0    0   892     0 1314   900  1  1 74 24
 0  1      0 1164392  84408 309040    0    0  2416     0 1713  1701  1  2 74 23
 0  1      0 1159656  84484 312704    0    0  3688     0 2023  2600  3  2 72 24
 0  1      0 1154984  84556 316100    0    0  3540     0 1985  2499  4  2 71 23
 1  0      0 1150312  84712 318936    0    0  2968     0 1853  1947  0  2 75 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1145384  84904 321260    0    0  2528     0 1774  1750  1  3 74 22
 0  1      0 1142192  85072 321976    0    0   948     0 1354   995  1  2 75 23
 0  1      0 1138992  85568 322772    0    0  1232     0 1380  1044  1  2 74 23
 0  1      0 1136496  85868 323084    0    0   676     0 1240   779  1  1 75 24
 0  1      0 1134448  86192 323372    0    0   604     0 1217   715  1  1 75 24
 0  1      0 1132720  86448 323728    0    0   608     0 1205   711  1  1 74 24
 0  1      0 1129968  86856 324204    0    0   812     0 1276   839  1  1 75 24
 0  1      0 1127920  87144 324460    0    0   648     0 1228   757  1  1 75 24
 0  1      0 1125360  87536 324884    0    0   760     0 1259   831  1  1 74 24
 0  1      0 1121904  87948 325152    0    0   732     0 1273   820  1  1 74 24
 0  1      0 1120048  88224 325556    0    0   640     0 1226   810  1  1 75 24
 1  0      0 1115248  88908 326164    0    0  1272     0 1420  1140  1  2 74 23
 0  1      0 1111792  89324 326564    0    0   860     0 1300   938  1  1 75 24
 0  1      0 1108912  89684 327156    0    0   952     0 1303   981  0  1 75 24
 0  1      0 1105072  90324 327536    0    0   940     0 1325   912  1  2 75 23
 1  0      0 1098736  91092 328060    0    0  1324     0 1464  1185  1  2 74 22
 0  1      0 1092720  91960 328416    0    0  1284     0 1437  1145  1  2 75 22
 0  1      0 1087728  92536 329540    0    0  1712   124 1543  1360  1  2 74 23
 0  1      0 1082672  93224 330756    0    0  1768     0 1533  1410  1  2 74 23
 0  1      0 1078576  93604 332008    0    0  1668     0 1491  1309  1  2 75 23
 0  1      0 1074544  94012 333164    0    0  1560     0 1461  1267  1  1 74 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0      0 1068976  94632 334720    0    0  2164     0 1629  1568  1  2 75 23
 0  2      0 1063856  95084 336104    0    0  1800  1820 1814  2197  1  3 62 35
 0  2      0 1060784  95324 336884    0    0  1084  1924 1625  1603  1  2 74 24
 0  2      0 1058928  95496 337664    0    0   952   148 1335  1422  0  1 75 24
 0  2      0 1055792  95744 338776    0    0  1336  1168 1624  1831  1  2 74 23
 0  2      0 1054320  95876 339188    0    0   604   936 1401  1027  0  1 67 32
 0  2      0 1053872  95928 339340    0    0   204  1360 1483   619  0  0 54 45
 0  2      0 1051504  96204 339948    0    0   856  1844 1616  1328  1  1 53 46
 0  1      0 1050736  96308 340184    0    0   300   576 1468   708  0  1 71 28
 0  1      0 1049328  96556 340412    0    0   528     8 1329   801  1  1 74 24
 0  1      0 1048688  96668 340572    0    0   264     0 1113   598  1  0 75 25
 0  1      0 1045744  97104 340884    0    0   780     0 1268   879  0  1 75 24
 0  1      0 1042480  97480 341596    0    0   980     0 1316  1013  1  1 74 24
 0  1      0 1041072  97656 341828    0    0   508     0 1177   729  0  0 75 24
 0  1      0 1039408  97904 342260    0    0   588     0 1209   779  0  1 75 24
 0  1      0 1034864  98448 342736    0    0  1060     0 1369  1036  1  2 75 23
 0  1      0 1032816  98780 343152    0    0   760     0 1250   867  1  1 74 24
 0  1      0 1031152  98984 343356    0    0   420     0 1161   646  1  1 75 24
 0  1      0 1029424  99184 343768    0    0   620     0 1221   710  1  1 75 24
 0  1      0 1026992  99504 344128    0    0   704     0 1239   755  1  1 75 24
 0  1      0 1024432  99792 344452    0    0   596     0 1220   724  1  1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 1016560 100840 344764    0    0  1364     0 1495  1178  1  3 75 21
 0  1      0 1007920 102004 345164    0    0  1512     0 1541  1265  1  4 74 21
 0  1      0 1000560 102940 345588    0    0  1388     0 1483  1232  1  3 74 22
 0  1      0 994416 103756 345996    0    0  1236     0 1432  1201  1  2 74 22
 1  0      0 986224 104984 346400    0    0  1568     0 1552  1616  5  4 71 21
 0  1      0 982896 105340 346792    0    0   780     0 1280   896  1  1 74 23
 0  1      0 980720 105600 347348    0    0   848     0 1272   948  0  1 75 24
 1  0      0 976304 106072 348168    0    0  1288     0 1423  1157  1  2 74 23
 0  1      0 972400 106676 349128    0    0  1540     0 1464  1199  1  1 74 23
 0  1      0 969840 107020 349804    0    0  1032     0 1325  1047  1  1 74 24
 0  1      0 967536 107372 350472    0    0  1004     0 1310  1051  3  1 73 24
 0  1      0 965040 107568 351228    0    0   944     0 1302   955  1  1 74 24
 0  1      0 960048 108148 352008    0    0  1360     0 1440  1426  5  2 71 22
 0  1      0 958128 108480 352424    0    0   804     4 1262  1031  3  1 72 24
 0  1      0 956336 108724 353064    0    0   840     0 1263   862  1  1 74 24
 0  1      0 954096 109032 353708    0    0   880     0 1284  1016  2  1 74 24
 1  0      0 950768 109412 354280    0    0  1008     0 1324   960  1  2 75 23
 1  1      0 948784 109656 354648    0    0   620     0 1221   813  1  1 74 24
 0  2      0 947568 109820 354892    0    0   424  2712 1576   854  0  1 70 28
 0  2      0 946480 109952 355100    0    0   344  1904 1492   829  0  1 67 31
 0  2      0 945776 110084 355240    0    0   264  2336 1514   706  1  1 65 34
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2      0 943984 110348 355792    0    0   808   548 1498  1245  0  1 67 31
 0  1      0 941424 110648 356444    0    0   984   296 1436  1304  1  1 70 29
 0  1      0 939248 111016 356892    0    0   784    28 1370   830  0  1 74 24
 0  1      0 936496 111372 357692    0    0  1128     0 1350   980  1  1 75 24
 0  1      0 935600 111500 357904    0    0   352     0 1138   587  0  0 75 25
 0  1      0 934000 111664 358216    0    0   448     0 1172   668  1  1 74 24
 0  1      0 931632 111948 358476    0    0   628     0 1240   746  1  1 75 24
 0  1      0 928816 112344 359304    0    0  1144     0 1351  1001  1  1 75 24
 0  1      0 925104 112828 360180    0    0  1372     0 1416  1105  1  1 75 23
 0  1      0 921584 113224 361212    0    0  1456     0 1438  1169  1  1 75 23
 1  0      0 916848 113788 362076    0    0  1452     0 1459  1253  1  2 74 23
 0  1      0 912688 114224 362660    0    0  1020     0 1349   957  1  2 75 23
 0  1      0 910320 114456 363244    0    0   844     0 1273   944  1  1 74 24
 0  1      0 907440 114756 364168    0    0  1176     0 1372  1120  2  1 73 24
 0  1      0 905840 115032 364368    0    0   520     0 1189   691  0  0 75 24
 0  1      0 904368 115320 364896    0    0   688     0 1223   841  1  0 74 24
 0  1      0 902704 115648 365180    0    0   664     0 1221   799  1  1 75 24
 0  1      0 900912 115968 365608    0    0   796     0 1260   884  1  1 75 24
 0  1      0 898992 116280 366044    0    0   688     0 1235   830  1  1 75 24
 0  1      0 897264 116572 366500    0    0   736     0 1239   855  1  1 74 24
 0  1      0 895536 116816 367072    0    0   876     0 1278   930  1  1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 892976 117148 367488    0    0   764     0 1260   865  1  1 75 23
 0  1      0 890864 117384 367932    0    0   652     0 1235   929  2  1 74 24
 0  1      0 889136 117580 368348    0    0   596     0 1206   692  1  1 75 24
 0  1      0 885808 118000 368676    0    0   756     0 1297   961  2  1 74 23
 0  1      0 883568 118276 369012    0    0   632     0 1229   904  2  1 74 23
 0  1      0 880752 118524 369784    0    0   996     0 1323  1888  2  1 73 23
 0  1      0 878320 118788 370472    0    0   936     0 1308  1004  1  1 75 24
 0  1      0 875760 119052 370888    0    0   688     0 1255   844  1  1 74 24
 0  1      0 871280 119192 372720    0    0  1936     0 1572  1498  1  2 75 23
 0  1      0 868528 119344 373724    0    0  1228     0 1381  1145  1  1 74 24
 0  1      0 865776 119480 374472    0    0   904     0 1311   955  1  1 74 24
 0  1      0 863024 119644 375192    0    0   836    40 1306   969  1  1 74 23
 0  1      0 860656 119808 375980    0    0   940     0 1325  1072  1  1 74 23
 1  1      0 857648 119992 376748    0    0  1020     0 1513  2110  2  2 73 23
 1  1      0 854704 120168 377592    0    0  1036     0 1577  1881  5  2 71 23
 0  1      0 852040 120528 378252    0    0   904     0 1401  1109  2  1 73 24
 1  2      0 849928 120876 378652    0    0   748  2152 1624  1913 11  2 61 25
 0  1      0 849368 120928 378736    0    0   152  1484 1763  3483 13  3 52 32
 0  1      0 848472 121040 378896    0    0   304     0 1573  1133  1  1 74 24
 1  1      0 847064 121192 379152    0    0   412     8 1402  1834  4  1 72 24
 0  1      0 845400 121364 379388    0    0   444     0 1490  3378 12  2 65 22
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1      0 843208 121556 379876    0    0   632     8 1394  3426 12  2 63 22
 1  1      0 840776 121792 380388    0    0   712     0 1448  3242  5  2 71 23
 0  1      0 838600 121956 380904    0    0   736     0 1480  3598 10  2 65 23
 0  1      0 837128 122116 381220    0    0   480     0 1322  1816  5  1 70 23
 0  1      0 834440 122384 381564    0    0   668     0 1388  1348  3  1 72 23
 1  1      0 831432 122704 382128    0    0   824     0 1394  1112  1  1 74 23
 0  1      0 828872 122944 382568    0    0   700     0 1258   851  1  1 74 23
 0  1      0 826568 123224 382900    0    0   592     0 1234   740  1  1 75 23
 0  1      0 824584 123476 383192    0    0   576     0 1347  1020  2  1 74 23
 0  1      0 821320 123792 383488    0    0   632     0 1394  1791  8  2 67 23
 0  1      0 816200 124356 384080    0    0  1060    28 1640  4422 20  5 55 19
 0  1      0 811784 124800 384724    0    0  1124     0 1533  1953  6  3 69 22
 0  1      0 808072 125160 385520    0    0  1148     0 1423  1270  1  2 74 23
 0  1      0 806216 125352 385804    0    0   524     0 1299  1015  1  1 74 24
 0  1      0 804248 125616 386220    0    0   648     0 1375  2226  7  1 68 23
 0  1      0 801880 125968 386752    0    0   872     4 1327  1045  1  1 74 23
 0  1      0 799640 126284 387184    0    0   780     0 1274   888  1  1 74 24
 2  0      0 797208 126608 388220    0    0  1308     0 1394  1075  1  1 75 24
 0  1      0 792824 127048 389276    0    0  1512     0 1476  1219  1  2 74 23
 0  1      0 787672 127500 390864    0    0  2040     0 1605  1475  1  2 75 23
 0  1      0 782488 127836 392704    0    0  2176     0 1629  1544  1  2 74 22
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1      0 776792 128300 394416    0    0  2156     0 1674  1559  1  2 74 22
 0  1      0 772952 128440 396452    0    0  2184     0 1665  1571  1  1 74 23
 0  1      0 767128 128644 400260    0    0  4012     0 2156  2524  1  2 74 23
 0  1      0 757592 129876 400796    0    0  1764     0 1753  1504  2  5 74 20
 1  1      0 753880 130136 402508    0    0  2020     0 1621  1506  1  1 74 23
 0  1      0 751000 130512 403016    0    0   868     0 1330   973  1  2 74 23
 0  1      0 747608 130684 404272    0    0  1444     0 1469  1267  1  1 75 23
 1  0      0 742744 130968 406436    0    0  2472     0 1779  1813  1  2 74 23
 1  1      0 738712 131128 408520    0    0  2180     0 1666  1565  1  2 74 23
 0  1      0 731032 131268 414160    0    0  5740    12 2584  4556  1  2 74 22
 0  1      0 727064 131332 416952    0    0  2884     8 1822  2695  1  1 74 24
 0  1      0 722200 131424 420192    0    0  3256     0 1927  2122  1  2 74 23
 0  1      0 720024 131544 420888    0    0   924     0 1324   950  1  1 74 24
 0  1      0 716952 131808 421508    0    0   876     0 1303   884  1  1 75 23
 0  1      0 715096 132024 421836    0    0   516     8 1202   713  1  1 74 24
 0  1      0 712472 132388 422220    0    0   816     0 1285   851  1  1 74 24
 0  1      0 710040 132608 422680    0    0   688     0 1247   803  1  1 74 24
 0  1      0 708696 132808 422888    0    0   424     0 1177   668  1  1 74 24
 0  1      0 705944 133064 423516    0    0   808     0 1284   854  1  1 74 24
 0  1      0 703896 133312 423812    0    0   608     8 1233   747  1  2 74 24
 0  1      0 701400 133564 424240    0    0   672     0 1247   822  1  1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 699480 133776 424572    0    0   556     0 1214   781  1  1 74 24
 0  1      0 696664 134148 425288    0    0  1020     0 1333  1023  1  1 74 24
 0  1      0 693784 134472 425644    0    0   724     0 1265   868  1  1 74 24
 0  1      0 691352 134692 426240    0    0   820     0 1282   906  1  1 74 24
 0  1      0 690008 134828 426580    0    0   496     0 1194   737  1  0 74 24
 1  1      0 688792 134912 426904    0    0   424     0 1168   722  1  0 74 24
 0  1      0 687192 135104 427188    0    0   464     0 1189   759  1  1 74 24
 0  1      0 685272 135312 427660    0    0   624     0 1228   821  1  1 74 24
 0  1      0 682328 135724 428064    0    0   848    20 1301   945  1  1 74 24
 0  1      0 679256 136108 428428    0    0   648     0 1245   841  1  1 74 24
 0  1      0 675544 136500 428852    0    0   812     0 1292   911  1  1 74 23
 0  1      0 671512 137208 429504    0    0  1396     0 1440  1155  1  2 75 23
 0  1      0 667992 137864 430548    0    0  1764     0 1521  1308  1  1 74 23
 1  1      0 664216 138904 431140    0    0  1620    56 1502  1393  1  2 74 24
 0  1      0 661464 139116 431744    0    0   832     0 1295   930  1  1 74 23
 1  1      0 657496 139492 432252    0    0   892     0 1347   993  1  1 74 23
 0  1      0 654168 139900 432864    0    0   976     0 1337  1029  1  2 74 23
 0  1      0 650520 140100 433888    0    0  1284     0 1414  1170  1  1 74 23
 0  2      0 647064 140476 434600    0    0  1028  2152 1651  1300  1  2 66 30
 0  2      0 644696 140520 435508    0    0   936  1128 1545  1451  1  1 71 27
 0  2      0 641368 140532 437060    0    0  1608   576 1651  2134  1  2 74 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2      0 639384 140604 437600    0    0   644   620 1403  1189  1  1 70 28
 0  2      0 637336 140692 438124    0    0   624   488 1381  1114  1  1 71 27
 0  2      0 635544 140764 438528    0    0   472   772 1589  1813  2  2 67 30
 0  2      0 634008 140828 439008    0    0   572   232 1316  1193  1  1 74 24
 0  2      0 631960 140892 439556    0    0   560   596 1352  1229  1  1 74 24
 0  2      0 630296 140920 440072    0    0   496   244 1275  1026  1  1 75 24
 0  2      0 628376 141024 440580    0    0   604   420 1392  1451  1  1 74 24
 0  2      0 627160 141112 440968    0    0   504   496 1321  1115  1  1 67 32
 0  1      0 625368 141268 441356    0    0   572   480 1397  1084  1  1 73 25
 0  1      0 623256 141456 441848    0    0   628     8 1289   849  1  1 75 24
 0  1      0 621144 141668 442384    0    0   740     0 1256   904  1  1 74 24
 0  1      0 618264 142172 442560    0    0   712     0 1262   861  1  2 74 23
 0  1      0 613336 142652 443032    0    0   960     0 1671  1823  2  3 74 21
 0  1      0 610840 142860 443844    0    0  1012     0 1496  3260 16  2 61 21
 0  1      0 609048 143004 444584    0    0   836     4 1458  1488  1  1 74 23
 0  1      0 607960 143064 445068    0    0   564     0 1189   697  1  1 75 24
 0  1      0 606616 143216 445324    0    0   368     0 1145   591  1  1 74 24
 1  0      0 603800 143548 445536    0    0   540     0 1207   693  1  2 75 23
 0  1      0 599640 144076 446028    0    0   964     0 1320   924  1  2 74 23
 1  0      0 594584 144736 446252    0    0   972     0 1347   943  1  3 75 22
 0  1      0 590872 145112 446828    0    0   924     0 1308   918  1  2 74 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 586712 145592 447368    0    0  1036     0 1353  1016  1  2 74 22
 0  1      0 585176 145740 447628    0    0   404     0 1151   679  1  1 74 24
 0  1      0 582744 146020 447892    0    0   548     0 1211   755  1  1 75 24
 0  1      0 579672 146328 448468    0    0   844     0 1288   936  1  1 74 24
 0  1      0 577304 146520 448888    0    0   656     0 1234   755  1  1 75 24
 0  1      0 574360 146828 449532    0    0   976     0 1318   928  1  1 74 24
 0  1      0 572120 146984 450124    0    0   716     0 1249   779  1  1 74 24
 0  1      0 568312 147424 450568    0    0   860     0 1309   862  1  2 75 23
 0  1      0 559956 148444 451112    0    0  1632     0 1545  1310  2  4 74 21
 0  1      0 558640 142800 452064    0    0  1608     0 1541  1313  2  4 74 21
 0  1      0 555760 140904 453892    0    0  2252     0 1648  1565  1  2 74 23
 0  1      0 554608 138056 455244    0    0  1676     0 1494  1285  1  2 74 23
 0  1      0 553556 137196 455628    0    0   656     0 1235   770  1  1 75 24
 1  1      0 552624 135148 456452    0    0   984     4 1323   950  1  1 74 23
 0  1      0 552496 133596 456780    0    0   636     0 1220   761  1  1 74 24
 0  1      0 551472 132792 457380    0    0   868     0 1276   920  1  1 75 24
 0  1      0 550640 131944 457956    0    0   720     0 1235   764  1  1 74 24
 0  1      0 550448 131016 458476    0    0   592     0 1194   698  1  0 74 24
 0  1      0 549872 130132 459088    0    0   788    12 1255   811  1  1 74 24
 1  1      0 549744 129196 459480    0    0   420     0 1154   653  1  1 75 25
 0  1      0 549360 128300 459968    0    0   592     0 1202   727  1  1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 548976 127448 460344    0    0   472     0 1171   660  1  1 75 24
 0  1      0 548336 126568 460816    0    0   724     0 1237   786  1  1 74 24
 0  1      0 546928 126700 461228    0    0   560    84 1212   737  1  1 74 24
 0  1      0 547024 123076 461656    0    0   864     0 1310   907  1  2 74 23
 0  1      0 545232 115292 461960    0    0  1832     0 1680  1537  2  7 74 17
 0  1      0 544080 111736 462864    0    0  1572     0 1489  1313  1  3 74 22
 2  0      0 543760 110556 463160    0    0   380     0 1150   675  1  1 74 24
 0  2      0 543352 108260 463348    0    0   740  3440 1595   991  1  2 67 30
 0  1      0 543292 107388 463744    0    0   520   680 1472   785  0  1 70 28
 0  1      0 542716 107540 463864    0    0   304     0 1122   612  1  0 75 24
 0  1      0 541564 104884 464820    0    0  1304     0 1405  1137  1  3 74 22
 0  1      0 541052 102168 465496    0    0  1160     0 1365  1064  1  2 74 23
 0  1      0 539132  99740 466360    0    0  1504     0 1466  1275  1  2 67 29
 0  1      0 538748  96940 467052    0    0  1144     0 1364  1078  1  2 74 23
 0  1      0 537660  95400 467572    0    0   996     0 1320  1013  1  2 74 23
 0  1      0 537396  92624 468036    0    0   964     0 1535  1837  2  2 73 22
 0  1      0 536756  90976 468256    0    0   720     0 1453  1985  5  2 71 23
 0  2      0 535924  89696 468448    0    0   912     0 1389  1062  1  2 62 36
 2  1      0 534580  88104 469632    0    0  1668     0 1469  1359  2  1 72 25
 0  1      0 533108  87400 470200    0    0   860     0 1295   945  1  1 74 23
 0  1      0 532980  85432 470468    0    0   652     0 1239   773  1  1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 532340  83688 470852    0    0   688     0 1247   851  1  1 74 23
 0  1      0 532020  81584 471188    0    0  1624     0 1486  1292  1  2 62 35
 0  1      0 531124  77008 471684    0    0  1244     0 1447  1314  2  3 73 22
 0  1      0 530932  75200 472064    0    0   660     0 1259  1023  2  1 73 24
 0  1      0 529780  74488 472368    0    0   628     0 1229   885  2  1 74 24
 0  1      0 529844  71588 473160    0    0  1044     0 1336  1028  1  1 74 23
 2  1      0 528628  70812 473528    0    0   776     0 1267   952  1  1 71 27
 0  1      0 527860  69876 473988    0    0   664     0 1238   973  2  1 74 24
 1  1      0 526564  68976 474616    0    0   912     0 1241  1607  8  1 67 24
 1  1      0 527012  66852 474904    0    0   444     0 1191   682  2  1 73 24
 0  1      0 525844  65200 475196    0    0   656     0 1596  1700  8  2 70 20
 0  1      0 525464  64340 475444    0    0   412     0 1430  1145  3  1 73 24
 0  1      0 524536  62160 476196    0    0  1024    36 1460  1656  7  2 69 22
 0  1      0 524664  59884 476500    0    0   488    36 1345   989  2  1 74 23
 1  0      0 523512  59012 476896    0    0   744    32 1461  1136  2  1 74 23
 0  1      0 522616  56180 477552    0    0  1076    44 1598  1632  3  2 74 21
 1  1      0 520944  54356 478356    0    0  1068     0 1303  1522  7  2 66 25
 0  1      0 520560  52380 479040    0    0   792     0 1448  1560  3  2 73 23
 1  1      0 517040  51280 479392    0    0   592    20 1441  1479  7  1 69 24
 0  1      0 514480  49292 480224    0    0  1040    32 1456  1439 15  2 61 22
 0  1      0 513264  48196 480436    0    0   500    24 1374  1140  2  1 73 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 511984  47400 480824    0    0   532    36 1309   911  1  1 72 25
 0  1      0 511152  45500 481296    0    0   524    64 1322   997  2  1 74 23
 0  1      0 509104  44836 481960    0    0   956    40 1356  1026  5  1 70 24
 0  1      0 508128  43028 482612    0    0  1092   140 1590  1580  2  2 68 28
 1  1      0 507636  42104 483468    0    0   968  4028 1720  1291 16  2 54 28
 0  1      0 506676  42148 483628    0    0   196     8 1350   655  5  0 70 25
 0  1      0 505844  41304 484064    0    0   588     0 1429   997  2  1 73 24
 0  1      0 504948  39072 484732    0    0   856    40 1383  1316  3  2 73 23
 0  1      0 503796  37372 485548    0    0  1076    40 1401  1549  4  2 71 22
 0  1      0 502452  36240 486136    0    0   924    52 1396  1674  5  2 71 23
 0  1      0 501876  34416 486736    0    0   768    12 1396  1846  8  3 68 21
 0  1      0 520372  32120 486584    0    0   796     8 1370  2233  7  3 68 22
 1  1      0 518980  28088 487148    0    0  1576    52 1656  2049  2  3 74 21
 0  1      0 518212  25440 488028    0    0  1296   144 1780  2383  2  3 73 22
 1  1      0 511244  24520 489152    0    0  1268   156 1496  6987 11  4 62 23
 0  1      0 505148  22600 490324    0    0  1180    28 1405  4216 24  3 51 22
 1  1      0 503660  21440 491280    0    0  1264   240 1613  2784 12  2 59 27
 0  1      0 502444  16728 492524    0    0  1816   160 1773  3944 12  5 64 19
 0  1      0 501740  12596 493460    0    0  1480   248 1745  2059  5  3 71 22
 0  1      0 500268  10280 494552    0    0  1584   168 1550  1881  2  2 73 23
 0  1      0 498348   9452 495244    0    0  1080   172 1433  1108  1  1 74 23
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1      0 498284   8300 495512    0    0   400   436 1430   832  1  1 75 23
 0  1      0 497812   7188 495740    0    0   452   204 1478  3440 13  3 62 23
 1  0      0 497044   4960 496608    0    0  1108   636 1725  1486  1  2 74 23
 0  1      0 496172   4052 497108    0    0   788   160 1444  1023  1  1 74 24
 0  1      0 495884   3116 497296    0    0   416   680 1559  1319  5  1 71 23
 0  2      0 496012   1436 497752    0    0   728   624 1524  1024  1  1 72 26
 0  3      0 494788    524 498188    0    0  1324   692 1715  1560  1  2 52 45
 0  2      0 494524    352 498360    0    0   876   920 1505  1369  0  1 52 46
20 17      0 506556    144 497888    0    0   180   124 2154  2167  0  0 11 88
 0 12      0 505980    136 498168    0    0   508   224 2617  1918  1  0 20 78
 0 18      0 516404    136 498168    0    0    20    20 2844  1537  0  0  7 92
 0 20      0 516916    124 497636    0    0    44    96 1754  1194  1  0  3 96
 0  1      0 516308    164 498004    0    0   664    36 1593  2134  3  1 67 28
 0  1      0 516052    132 498172    0    0   288   180 1387   988  1  0 70 28
 1  4      0 518756    136 495516    0    0   672   124 1533  1248  1  1 50 48
 0  4      0 525668    136 489124    0    0   700   168 1451  1117  2  0 57 41
 0  3      0 544932    148 469800    0    0   544   244 1572  1661  1  2 66 31
 0  7      0 544676    152 470000    0    0   544     0 1213   723  1  1 68 30
 0  1      0 546324    224 468024    0    0   816   180 1567  1566  2  1 64 33
 0  2      0 551380    204 462944    0    0   528    80 1325   963  1  1 59 39
 0  2      0 552148    192 463636    0    0  1028    80 1373  1082  1  1 63 35
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  1      0 551636    152 464084    0    0  1068   320 1484  1542  1  2 63 34
 0  2      0 552212    164 463392    0    0   532     4 1200   678  1  0 70 29
 3  0      0 551124    364 463872    0    0  1048   224 1602  1900  8  2 54 36
 0  2      0 549396    236 464816    0    0  1596   400 1555  1811  3  2 56 39
 0  2      0 550804    208 464980    0    0   452   180 1460  2644  3  1 70 26
 0  6      0 550164    152 465444    0    0  1088   220 1649  1829  2  1 53 43
 0  1      0 549972    156 465644    0    0   412    72 1344   940  1  1 74 25
 0  1      0 549652    196 465808    0    0   556   220 1637  1444  1  1 68 30
 0  1      0 549396    180 465960    0    0   400   136 1466  1893  3  1 66 30
 0  0      0 638292    160 466048    0    0   300   144 1228   858  1  1 80 18
 0  0      0 638292    160 466048    0    0     0     0 1308  1326  2  0 98  0
 1  0      0 638300    160 466048    0    0     0     0 1277   964  1  0 99  0
 0  0      0 638300    160 466048    0    0     0     0 1211   946  1  0 98  0
 0  0      0 638364    168 466040    0    0     8     0 1349  1260  2  0 97  0
 0  1      0 636940    268 466144    0    0   244     0 1224  5457  3  3 91  4
 3  0      0 630100    392 467448    0    0  1136     0 1169  1786 14  2 74 10
 0  0      0 626004    416 468172    0    0   572     0 1163  2152 15  1 82  2
 2  0      0 625876    416 468240    0    0     0     0 1220  2859 16  1 84  0
 2  0      0 625812    416 468240    0    0     0     8 1343  3184 15  1 84  0
 0  0      0 625932    436 468220    0    0   324   416 1529  1656  1  1 80 18
 2  1      0 625868    444 468280    0    0    12     0 1249  1111  2  0 98  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1      0 615884    468 468596    0    0    40     4 1241  3132 26  3 70  1
 0  0      0 614540    476 468792    0    0     8     4 1234  2539 13  1 85  1
 0  0      0 614540    476 468792    0    0     0     0 1123   619  0  0 99  0
 0  0      0 614540    476 468792    0    0     0     0 1057   612  0  0 100  0
 0  0      0 614548    476 468792    0    0     0     0 1058   675  1  0 99  0
 0  0      0 614292    484 468852    0    0    20    36 1064  2015 10  1 88  1
 0  0      0 614356    484 468852    0    0     0     0 1232   646  1  0 99  0
 0  0      0 614228    484 468852    0    0     0     0 1119  1103  9  1 90  0
 0  0      0 614296    484 468852    0    0     0     0 1148   569  1  0 99  0
 0  0      0 614488    500 468904    0    0    84    16 1135   486  0  0 98  1
 0  0      0 614556    504 468968    0    0     4     0 1050   342  1  0 99  0
 0  0      0 614556    504 468968    0    0     0     0 1047   343  0  0 100  0
 0  0      0 614644    504 468968    0    0     0     0 1043   316  0  0 100  0
 0  0      0 614652    504 468968    0    0     0     0 1036   324  0  0 100  0
 0  0      0 614752    504 468968    0    0     0     0 1037   336  0  0 100  0
 0  0      0 614752    504 468968    0    0     0     0 1036   379  0  0 100  0
 0  0      0 614752    504 468968    0    0     0     0 1036   383  0  0 100  0
 0  0      0 614752    504 468968    0    0     0     0 1037   376  0  0 100  0
 0  0      0 614956    504 468968    0    0     0     0 1037   308  0  0 100  0
 0  0      0 614956    504 468968    0    0     0     0 1037   322  0  0 100  0
 0  0      0 615024    504 468968    0    0     0     0 1037   302  0  0 100  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 615024    504 468968    0    0     0     0 1036   308  0  0 100  0
 0  0      0 615164    504 468968    0    0     0     0 1037   310  0  0 100  0
 0  0      0 615164    504 468968    0    0     0     0 1036   338  0  0 100  0
 0  0      0 615164    504 468968    0    0     0     0 1037   335  0  0 100  0
 0  0      0 615164    504 468968    0    0     0     0 1036   321  0  0 100  0
 0  0      0 615292    504 468968    0    0     0     0 1037   313  0  0 100  0
 0  0      0 615292    504 468968    0    0     0     0 1036   316  0  0 100  0
 1  0      0 615356    504 468968    0    0     0     8 1038   326  0  0 100  0
 0  0      0 615356    504 468968    0    0     0     0 1037   331  0  0 100  0
 0  0      0 615420    504 468968    0    0     0     0 1036   306  0  0 100  0
 0  0      0 615420    504 468968    0    0     0     0 1185   674  1  0 100  0
 0  0      0 615420    504 468968    0    0     0     0 1297   894  1  0 99  0
 0  0      0 615052    604 468936    0    0   100   528 1392  3551 20  1 76  2
 0  0      0 615052    604 468936    0    0     0     0 1278  2523 13  1 86  0
 0  0      0 615052    604 468936    0    0     0     0 1272  2354 10  1 89  0
 0  0      0 615052    604 468936    0    0     0     0 1225  2248 14  1 85  0
 1  0      0 615052    604 468936    0    0     0     0 1400  3337 27  2 71  0
 0  0      0 615060    628 468912    0    0    24     0 1087  3809  6  1 92  1
 0  0      0 615060    628 468912    0    0     0     0 1044   450  0  0 100  0
 0  0      0 615060    628 468912    0    0     0     0 1161  2010  9  1 91  0
 1  0      0 615060    628 468912    0    0     0     0 1129  1299  6  0 93  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 615060    628 468912    0    0     0     0 1229  3173 19  1 79  0
 0  0      0 615052    656 468884    0    0    28    92 1107   766  0  0 98  1
 0  0      0 615052    656 468884    0    0     0     0 1261   715  0  0 99  0
 0  0      0 615052    656 468884    0    0     0     0 1189   634  0  0 100  0
 0  0      0 615116    660 468880    0    0     4     0 1081   493  0  0 99  0
 0  0      0 615116    660 468880    0    0     0     0 1056   391  0  0 100  0
 0  0      0 615188    660 468880    0    0     0     8 1071   475  0  0 100  0
 0  0      0 615236    660 468880    0    0     0     0 1324  2315  9  1 89  0
 0  0      0 615236    660 468880    0    0     0     0 1252   692  1  0 99  0
 0  0      0 615012    660 468880    0    0     0     0 1080   897  4  0 96  0
 2  0      0 614820    668 469076    0    0   212     0 1249  2403  8  1 90  1
 0  0      0 614820    668 469076    0    0     0     0 1234   846  2  0 98  0
 0  0      0 614828    668 469076    0    0     0     0 1097   473  0  0 100  0
 0  0      0 614828    668 469076    0    0     0     0 1165   490  0  0 100  0
 0  0      0 614892    668 469076    0    0     0     0 1209   611  1  0 99  0
 1  0      0 614892    668 469076    0    0     0     0 1068   665  3  0 97  0
 0  0      0 614988    668 469076    0    0     0     0 1061   493  1  0 99  0
 0  0      0 614988    668 469076    0    0     0     0 1259   890  1  0 99  0
 0  0      0 615052    668 469076    0    0     0     0 1050   614  0  0 99  0
 0  0      0 615052    668 469076    0    0     0     0 1036   367  0  0 100  0
 0  0      0 615148    668 469076    0    0     0     0 1037   367  0  0 100  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 615148    668 469076    0    0     0     0 1036   374  0  0 100  0
 0  0      0 615148    668 469076    0    0     0     0 1037   376  0  0 100  0
 0  0      0 615148    668 469076    0    0     0     0 1036   354  0  0 100  0
 0  0      0 615292    668 469076    0    0     0     0 1037   294  0  0 100  0
 0  0      0 615292    668 469076    0    0     0     0 1037   296  0  0 100  0
 0  0      0 615292    668 469076    0    0     0   100 1061   307  0  0 100  0
 0  0      0 615292    668 469076    0    0     0     0 1036   345  0  0 100  0
 0  0      0 615308    668 469076    0    0     0     0 1037   298  0  0 100  0
 0  0      0 615308    668 469076    0    0     0     0 1044   361  0  0 100  0
 0  0      0 615372    668 469076    0    0     0     0 1105   416  0  0 100  0
 0  0      0 615372    668 469076    0    0     0     0 1188   583  1  0 99  0
 0  0      0 615356    668 469076    0    0     0     0 1333  1283  4  0 96  0
 0  0      0 615356    668 469076    0    0     0     0 1200   617  0  0 100  0
 0  0      0 615356    668 469076    0    0     0     0 1098   491  0  0 100  0
 0  0      0 615332    668 469076    0    0     0     0 1209  1861  7  1 92  0
 0  0      0 615396    668 469076    0    0     0     0 1213   584  0  0 100  0
 0  0      0 615396    668 469076    0    0     0     0 1089   397  0  0 100  0
 0  0      0 615396    668 469076    0    0     0     0 1057   325  0  0 100  0
 0  0      0 615140    676 469068    0    0     8     0 1301  2154 10  1 89  0
 1  0      0 615140    676 469068    0    0     0     0 1157   482  0  0 100  0
 0  0      0 615140    676 469068    0    0     0     4 1182   942  1  0 99  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 615140    676 469068    0    0     0     0 1280  1122  1  0 99  0
 0  0      0 615140    676 469068    0    0     0     0 1046   447  0  0 100  0
 1  0      0 615212    676 469068    0    0     0     0 1043   452  0  0 100  0
 0  0      0 615212    676 469068    0    0     0     0 1039   362  0  0 100  0
 0  0      0 615212    676 469068    0    0     0     0 1038   411  0  0 100  0
 0  0      0 615212    676 469068    0    0     0     0 1044   530  0  0 100  0
 0  0      0 615244    676 469068    0    0     0     0 1042   432  0  0 100  0
 0  0      0 615244    676 469068    0    0     0     0 1044   449  0  0 100  0
 0  0      0 615236    676 469068    0    0     0     0 1039   339  0  0 100  0
 0  0      0 615236    676 469068    0    0     0     0 1038   351  0  0 100  0
 0  0      0 615300    676 469068    0    0     0     0 1043   443  0  0 100  0

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: rsync out of memory 2.6.8-rc2
@ 2004-07-19  9:19 Klaus Dittrich
  2004-07-19 11:20 ` dentry cache leak? " bert hubert
  0 siblings, 1 reply; 8+ messages in thread
From: Klaus Dittrich @ 2004-07-19  9:19 UTC (permalink / raw)
  To: linux mailing-list; +Cc: gilbertd, nickpiggin, kladit

Nick your mail got lost here, sorry.

I did rsync -av xeon2:/disc1 . on the target system.
xeon2 is the system that gets out of memory.
/disc1 is 60GB and contains several thousand files.
I think this is a common backup situation.

I found out I could trigger the memory outage using du -s /disc1 too.

-- 
Klaus

^ permalink raw reply	[flat|nested] 8+ messages in thread

* dentry cache leak? Re: rsync out of memory 2.6.8-rc2
  2004-07-19  9:19 rsync out of memory 2.6.8-rc2 Klaus Dittrich
@ 2004-07-19 11:20 ` bert hubert
       [not found]   ` <20040719113228.GA15295@outpost.ds9a.nl>
  0 siblings, 1 reply; 8+ messages in thread
From: bert hubert @ 2004-07-19 11:20 UTC (permalink / raw)
  To: Klaus Dittrich; +Cc: linux mailing-list, gilbertd, nickpiggin

On Mon, Jul 19, 2004 at 11:19:43AM +0200, Klaus Dittrich wrote:

> I found out I could trigger the memory outage using du -s /disc1 too.

Including crashing and/or running out of swap? That would indicate that the
dentry cache is not cleaning itself up, or that something is wrong with
reference counting.

Can you run 'cat fs/dentry-state' before and after the du -s? (assuming
there is an 'after'. Also, which fs is /disc1 on? any messages in dmesg?

dentry-state
------------

Status of  the  directory  cache.  Since  directory  entries  are
dynamically allocated and deallocated, this file indicates the current
status. It holds six values, in which the last two are not used and are
always zero. The others are listed in table 2-1.


Table 2-1: Status files of the directory cache 
..............................................................................
 File       Content                                                            
 nr_dentry  Almost always zero                                                 
 nr_unused  Number of unused cache entries                                     
 age_limit  
            in seconds after the entry may be reclaimed, when memory is
short 
 want_pages internally                                                         


-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: dentry cache leak? Re: rsync out of memory 2.6.8-rc2
       [not found]   ` <20040719113228.GA15295@outpost.ds9a.nl>
@ 2004-07-19 12:56     ` Klaus Dittrich
  2004-07-26  1:06       ` Andrew Morton
  0 siblings, 1 reply; 8+ messages in thread
From: Klaus Dittrich @ 2004-07-19 12:56 UTC (permalink / raw)
  To: bert hubert; +Cc: Klaus Dittrich, linux mailing-list

bert hubert wrote:

>/proc/fs/dentry-state of course, sorry
>
>On Mon, Jul 19, 2004 at 01:20:47PM +0200, bert hubert wrote:
>  
>
>>On Mon, Jul 19, 2004 at 11:19:43AM +0200, Klaus Dittrich wrote:
>>
>>    
>>
>>>I found out I could trigger the memory outage using du -s /disc1 too.
>>>      
>>>
>>Including crashing and/or running out of swap? That would indicate that the
>>dentry cache is not cleaning itself up, or that something is wrong with
>>reference counting.
>>
>>Can you run 'cat fs/dentry-state' before and after the du -s? (assuming
>>there is an 'after'. Also, which fs is /disc1 on? any messages in dmesg?
>>
>>dentry-state
>>------------
>>
>>Status of  the  directory  cache.  Since  directory  entries  are
>>dynamically allocated and deallocated, this file indicates the current
>>status. It holds six values, in which the last two are not used and are
>>always zero. The others are listed in table 2-1.
>>
>>
>>Table 2-1: Status files of the directory cache 
>>..............................................................................
>> File       Content                                                            
>> nr_dentry  Almost always zero                                                 
>> nr_unused  Number of unused cache entries                                     
>> age_limit  
>>            in seconds after the entry may be reclaimed, when memory is
>>short 
>> want_pages internally                                                         
>>
>>
>>-- 
>>http://www.PowerDNS.com      Open source, database driven DNS Software 
>>http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>>
>>    
>>
>
>  
>
The fs is ext2.cat /proc/sys/fs/dentry-state
Output of  cat /proc/sys/fs/dentry-state before and after processes got 
killed.
891083  888395  45      0       0       0
1142933 1085759 45      0       0       0
--
Klaus

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: dentry cache leak? Re: rsync out of memory 2.6.8-rc2
  2004-07-19 12:56     ` Klaus Dittrich
@ 2004-07-26  1:06       ` Andrew Morton
  0 siblings, 0 replies; 8+ messages in thread
From: Andrew Morton @ 2004-07-26  1:06 UTC (permalink / raw)
  To: Klaus Dittrich; +Cc: ahu, kladit, linux-kernel

kladit@t-online.de (Klaus Dittrich) wrote:
>
> The fs is ext2.cat /proc/sys/fs/dentry-state
>  Output of  cat /proc/sys/fs/dentry-state before and after processes got 
>  killed.
>  891083  888395  45      0       0       0
>  1142933 1085759 45      0       0       0

This bug is probably unique to yourself and a couple of other people.  If
it was hitting everyone, all the machines in the world would be going oom
during the nightly updatedb run.

Can you try a different compiler version?  And try disabling any "unusual"
config options, such as filesystem quotas, extended attributes, etc?

Can you narrow the onset of the problem down to any particular kernel
snapshot?

Try disabling laptop_mode, if it's set.

Carefully review the logs for any oopses: oopsing while holding
shrinker_sem will do this for sure.


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2004-07-26  1:08 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-07-19  9:19 rsync out of memory 2.6.8-rc2 Klaus Dittrich
2004-07-19 11:20 ` dentry cache leak? " bert hubert
     [not found]   ` <20040719113228.GA15295@outpost.ds9a.nl>
2004-07-19 12:56     ` Klaus Dittrich
2004-07-26  1:06       ` Andrew Morton
  -- strict thread matches above, loose matches on Subject: below --
2004-07-18 21:52 Klaus Dittrich
2004-07-19  7:48 ` Nick Piggin
2004-07-19  7:52   ` bert hubert
2004-07-19  8:47     ` Klaus Dittrich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox