Very high memory usage with KVM

Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1944 bytes --]

Hi all!

I have an installation with Ubuntu Hardy Heron server amd64 with KVM-62
from Ubuntu repositories installed on an HP Proliant DL380 G5 with two
Xeon E5405 quadcore processors and 16 GiB of RAM which has six VMs with
the following configuration of memory:

Hostname       |      RAM
===============+===============
Ganimedes      |    2 GiB
Os             |    1 GiB
Aprender       |    2 GiB
Aps0           |    2 GiB
Aps2           |    4 GiB
Ratatoskr      |    4 GiB
===============+===============
TOTAL          |   15 GiB


Initially the host was created with a swap partition of 1 GiB (more 1
GiB than was free for use of host) but this amount with the time
remained short and I had to add a LV of 7 GiB to be used with swap,
being now a total of 8 GiB of swap of which at this moment I have only a
9% free. Is 'normal' this use of memory?

root@ss02:~# ps -e --sort -rss -Ho user,start_time,pid,pcpu,pmem,rss,size,vsz,args
USER     START   PID %CPU %MEM   RSS    SZ    VSZ COMMAND
[...]
root     Jul06 27471 52.3 24.4 4023232 4292200 4350296   kvm <ratatoskr>
root     Jul24  9955  137 23.8 3923620 4308592 4350308   kvm <aps2>
root     Jul06  8751  5.8  8.3 1368228 2171808 2229888   kvm <aps0>
root     Jul07  8565  2.7  5.2 862844 2204704 2246416   kvm <aprender>
root     Apr22  7842  0.6  3.6 600072 2172056 2230136   kvm <ganimedes>
root     Jul01  7944  0.6  2.0 334860 1119916 1177996   kvm <os>

root@ss02:~# free
             total       used       free     shared    buffers     cached
Mem:      16463388   16377844      85544          0     894216      66328
-/+ buffers/cache:   15417300    1046088
Swap:      8319948    7621916     698032


Updating to KVM-84 or superior can improve this situation?

Thanks in advance.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Very high memory usage with KVM

[Avi Kivity]

On 07/25/2009 08:43 PM, Daniel Bareiro wrote:
> Hi all!
>
> I have an installation with Ubuntu Hardy Heron server amd64 with KVM-62
> from Ubuntu repositories installed on an HP Proliant DL380 G5 with two
> Xeon E5405 quadcore processors and 16 GiB of RAM which has six VMs with
> the following configuration of memory:
>
> Hostname       |      RAM
> ===============+===============
> Ganimedes      |    2 GiB
> Os             |    1 GiB
> Aprender       |    2 GiB
> Aps0           |    2 GiB
> Aps2           |    4 GiB
> Ratatoskr      |    4 GiB
> ===============+===============
> TOTAL          |   15 GiB
>
>
> Initially the host was created with a swap partition of 1 GiB (more 1
> GiB than was free for use of host) but this amount with the time
> remained short and I had to add a LV of 7 GiB to be used with swap,
> being now a total of 8 GiB of swap of which at this moment I have only a
> 9% free. Is 'normal' this use of memory?
>
> root@ss02:~# ps -e --sort -rss -Ho user,start_time,pid,pcpu,pmem,rss,size,vsz,args
> USER     START   PID %CPU %MEM   RSS    SZ    VSZ COMMAND
> [...]
> root     Jul06 27471 52.3 24.4 4023232 4292200 4350296   kvm<ratatoskr>
> root     Jul24  9955  137 23.8 3923620 4308592 4350308   kvm<aps2>
> root     Jul06  8751  5.8  8.3 1368228 2171808 2229888   kvm<aps0>
> root     Jul07  8565  2.7  5.2 862844 2204704 2246416   kvm<aprender>
> root     Apr22  7842  0.6  3.6 600072 2172056 2230136   kvm<ganimedes>
> root     Jul01  7944  0.6  2.0 334860 1119916 1177996   kvm<os>
>
> root@ss02:~# free
>               total       used       free     shared    buffers     cached
> Mem:      16463388   16377844      85544          0     894216      66328
> -/+ buffers/cache:   15417300    1046088
> Swap:      8319948    7621916     698032
>
>
> Updating to KVM-84 or superior can improve this situation?
>    

What is the storage configuration?  Are you using qcow2?  What are the 
image logical and physical sizes?

What is the host kernel (uname -a)?

-- 
error compiling committee.c: too many arguments to function

Re: Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 9684 bytes --]

Hi Avi.

On Sunday, 26 July 2009 14:31:57 +0300,
Avi Kivity wrote:

> >I have an installation with Ubuntu Hardy Heron server amd64 with KVM-62
> >from Ubuntu repositories installed on an HP Proliant DL380 G5 with two
> >Xeon E5405 quadcore processors and 16 GiB of RAM which has six VMs with
> >the following configuration of memory:
> >
> >Hostname       |      RAM
> >===============+===============
> >Ganimedes      |    2 GiB
> >Os             |    1 GiB
> >Aprender       |    2 GiB
> >Aps0           |    2 GiB
> >Aps2           |    4 GiB
> >Ratatoskr      |    4 GiB
> >===============+===============
> >TOTAL          |   15 GiB
> >
> >
> >Initially the host was created with a swap partition of 1 GiB (more 1
> >GiB than was free for use of host) but this amount with the time
> >remained short and I had to add a LV of 7 GiB to be used with swap,
> >being now a total of 8 GiB of swap of which at this moment I have only a
> >9% free. Is 'normal' this use of memory?
> >
> >root@ss02:~# ps -e --sort -rss -Ho user,start_time,pid,pcpu,pmem,rss,size,vsz,args
> >USER     START   PID %CPU %MEM   RSS    SZ    VSZ COMMAND
> >[...]
> >root     Jul06 27471 52.3 24.4 4023232 4292200 4350296   kvm<ratatoskr>
> >root     Jul24  9955  137 23.8 3923620 4308592 4350308   kvm<aps2>
> >root     Jul06  8751  5.8  8.3 1368228 2171808 2229888   kvm<aps0>
> >root     Jul07  8565  2.7  5.2 862844 2204704 2246416   kvm<aprender>
> >root     Apr22  7842  0.6  3.6 600072 2172056 2230136   kvm<ganimedes>
> >root     Jul01  7944  0.6  2.0 334860 1119916 1177996   kvm<os>
> >
> >root@ss02:~# free
> >              total       used       free     shared    buffers     cached
> >Mem:      16463388   16377844      85544          0     894216      66328
> >-/+ buffers/cache:   15417300    1046088
> >Swap:      8319948    7621916     698032
> >
> >
> >Updating to KVM-84 or superior can improve this situation?
 
> What is the storage configuration? Are you using qcow2?

The host machine has 8 x 300 GiB SAS disk in RAID 5 by hardware (7 disks
with 1 spare) on 1 logicaldrive. The partitioning scheme of host is the
following one:


root@ss02:~# fdisk -l /dev/cciss/c0d0

Disk /dev/cciss/c0d0: 1799.7 GB, 1799797127168 bytes
255 heads, 63 sectors/track, 218812 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000af3c3

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1               1         122      979933+  82  Linux swap / Solaris
/dev/cciss/c0d0p2   *         123        1338     9767520   83  Linux
/dev/cciss/c0d0p3            1339      218812  1746859905   8e  Linux LVM


I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical
volume that maintains the logical volumes that are used for VM's disks:


root@ss02:~# pvs
  PV                VG   Fmt  Attr PSize PFree
  /dev/cciss/c0d0p3 vm   lvm2 a-   1,63T 1,13T


> What are the image logical and physical sizes?

The disks for the VMs have these sizes:

  aprender-raiz   vm   -wi-ao   8,00G
  aprender-space  vm   -wi-ao  20,00G

  aps0-raiz       vm   -wi-ao   7,00G
  aps0-space      vm   -wi-ao  10,00G

  aps2-cache      vm   -wi-ao  20,00G
  aps2-index      vm   -wi-ao  10,00G
  aps2-raiz       vm   -wi-ao   7,00G
  aps2-space      vm   -wi-ao  10,00G

  ganimedes-raiz  vm   -wi-ao   5,00G
  ganimedes-space vm   -wi-ao  10,00G

  os-disk         vm   -wi-ao   6,00G
  os-mailbox      vm   -wi-ao 150,00G
  os-space        vm   -wi-ao  10,00G

  ratatoskr-raiz  vm   -wi-ao   8,00G
  ratatoskr-space vm   -wi-ao 200,00G

With respect to the internal partitioning scheme for each one of the
VMs, it is the following one:

* Aprender:

aprender:~# fdisk -l /dev/hda

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         486     3903763+  82  Linux swap / Solaris
/dev/hda2             487        1044     4482135   83  Linux
aprender:~#
aprender:~#
aprender:~# fdisk -l /dev/hdb

Disk /dev/hdb: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1        2610    20964793+  83  Linux


* Aps0:

[root@aps:~]
$fdisk -l /dev/hda

Disco /dev/hda: 7516 MB, 7516192768 bytes
255 heads, 63 sectors/track, 913 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00039c2a

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         365     2931831   82  Linux swap / Solaris
/dev/hda2             366         913     4401810   83  Linux
[root@aps:~]
$
[root@aps:~]
$fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00087cc1

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux


* Aps2

aps2:~# fdisk -l /dev/hda

Disco /dev/hda: 7516 MB, 7516192768 bytes
255 heads, 63 sectors/track, 913 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0004b793

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         365     2931831   82  Linux swap /
Solaris
/dev/hda2             366         913     4401810   83  Linux
aps2:~#
aps2:~#
aps2:~# fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0009491d

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux
aps2:~#
aps2:~#
aps2:~# fdisk -l /dev/hdc

Disco /dev/hdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdc1               1        1305    10482381   83  Linux
aps2:~#
aps2:~#
aps2:~# fdisk -l /dev/hdd

Disco /dev/hdd: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdd1               1        2610    20964793+  83  Linux


* Ganimedes:

[root@ganimedes:~]
$fdisk -l /dev/hda

Disco /dev/hda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0002cad1

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         122      979933+  82  Linux swap /
Solaris
/dev/hda2             123         652     4257225   83  Linux
[root@ganimedes:~]
$
[root@ganimedes:~]
$fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0007e07e

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux


* Os:

os:~# fdisk -l /dev/hda

Disco /dev/hda: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a78a9

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         243     1951866   82  Linux swap /
Solaris
/dev/hda2             244         783     4337550   83  Linux
os:~#
os:~#
os:~# fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000dffb2

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux
os:~#
os:~#
os:~# fdisk -l /dev/hdc

Disco /dev/hdc: 161.0 GB, 161061273600 bytes
255 heads, 63 sectors/track, 19581 cylinders
Units = cilindros of 16065 * 512 = 8225280 bytes
Disk identifier: 0x90c7dc67

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdc1               1       19581   157284351   83  Linux



* Ratatosk:

ratatoskr:~# fdisk -l /dev/hda

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         486     3903763+  82  Linux swap /
Solaris
/dev/hda2             487        1044     4482135   83  Linux
ratatoskr:~#
ratatoskr:~#
ratatoskr:~# fdisk -l /dev/hdb

Disk /dev/hdb: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1       26108   209712478+  83  Linux


> What is the host kernel (uname -a)?

root@ss02:~# uname -a
Linux ss02 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux



Thanks for your reply.
 
Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Very high memory usage with KVM

[Avi Kivity]

On 07/26/2009 05:56 PM, Daniel Bareiro wrote:
>
>> What is the storage configuration? Are you using qcow2?
>>      
>
> I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical
> volume that maintains the logical volumes that are used for VM's disks:
>
>    

In this case there should be no excessive memory usage.  qcow2 could use 
extra memory, especially on older qemu-kvm versions (or images created 
with older qemu-img versions).

>> What is the host kernel (uname -a)?
>>      
>
> root@ss02:~# uname -a
> Linux ss02 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux
>
>    

kvm memory management with pre 2.6.27 host kernels is pretty weak.  
Using a newer host kernel (and newer kvm) may solve this problem.

-- 
error compiling committee.c: too many arguments to function

Re: Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

Avi

On Sunday, 26 July 2009 18:11:27 +0300,
Avi Kivity wrote:

> >>What is the storage configuration? Are you using qcow2?

> >I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a
> >physical volume that maintains the logical volumes that are used for
> >VM's disks:
 
> In this case there should be no excessive memory usage.  qcow2 could
> use extra memory, especially on older qemu-kvm versions (or images
> created with older qemu-img versions).
 
> >>What is the host kernel (uname -a)?

> >root@ss02:~# uname -a
> >Linux ss02 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux

> kvm memory management with pre 2.6.27 host kernels is pretty weak.
> Using a newer host kernel (and newer kvm) may solve this problem.

Initially I am going to see how it improves the situation upgrading to
KVM-84 of backports of Hardy Heron, since at the moment last kernel
available for Hardy is the one I has commented.

I would like to know if the newest versions of KVM published in the
official site of the project solve a bug recently reported in Ubuntu
Launchpad [1]. Also I was observing errors of type 'to swapper Not
tainted' or 'java Not tainted' in the VM (aps2, with a high rate of I/O)
and that I've commented in a previous message sent to the list [2]. I
would want to know if you could indicate to me if this can be due to KVM
bug that would be solved in a later version.

Thanks for so quick reply.

Regards,
Daniel

[1] https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/359447
[2] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/37631
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Very high memory usage with KVM

[Avi Kivity]

On 07/26/2009 06:50 PM, Daniel Bareiro wrote:
>> kvm memory management with pre 2.6.27 host kernels is pretty weak.
>> Using a newer host kernel (and newer kvm) may solve this problem.
>>      
>
> Initially I am going to see how it improves the situation upgrading to
> KVM-84 of backports of Hardy Heron, since at the moment last kernel
> available for Hardy is the one I has commented.
>
> I would like to know if the newest versions of KVM published in the
> official site of the project solve a bug recently reported in Ubuntu
> Launchpad [1]. Also I was observing errors of type 'to swapper Not
> tainted' or 'java Not tainted' in the VM (aps2, with a high rate of I/O)
> and that I've commented in a previous message sent to the list [2]. I
> would want to know if you could indicate to me if this can be due to KVM
> bug that would be solved in a later version.
>
>    

My guess is that it is due to poor swapping with pre-2.6.27 hosts. 15 GB 
used out of 16GB total is just 6% reserve, which may be a bit too low.  
With a 2.6.27 host some small amount of memory would be swapped out, 
before that you'd see thrashing.  Another way to check is to drop one 
guest (or reduce total memory needed by 1GB) and see if you get the same 
results or if things improve.

-- 
error compiling committee.c: too many arguments to function

Re: Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 6203 bytes --]

Hi Avi.

On Sunday, 26 July 2009 19:19:39 +0300,
Avi Kivity wrote:

> >>kvm memory management with pre 2.6.27 host kernels is pretty weak.
> >>Using a newer host kernel (and newer kvm) may solve this problem.
> >
> >Initially I am going to see how it improves the situation upgrading
> >to KVM-84 of backports of Hardy Heron, since at the moment last
> >kernel available for Hardy is the one I has commented.
> >
> >I would like to know if the newest versions of KVM published in the
> >official site of the project solve a bug recently reported in Ubuntu
> >Launchpad [1]. Also I was observing errors of type 'to swapper Not
> >tainted' or 'java Not tainted' in the VM (aps2, with a high rate of
> >I/O) and that I've commented in a previous message sent to the list
> >[2]. I would want to know if you could indicate to me if this can be
> >due to KVM bug that would be solved in a later version.

> My guess is that it is due to poor swapping with pre-2.6.27 hosts.
> 15 GB used out of 16GB total is just 6% reserve, which may be a bit
> too low.  With a 2.6.27 host some small amount of memory would be
> swapped out, before that you'd see thrashing.  Another way to check
> is to drop one guest (or reduce total memory needed by 1GB) and see
> if you get the same results or if things improve.

I was testing in another equipment of similar characteristics with 4 VMs of
3.5 GiB of RAM on LVM and rather less the 16 GB of RAM in use in the host
machine and 4 GIB available of the 8 GIB of swap, and after to update to
KVM-88 the use of virtual memory improved enough.

At the moment these are the statistics of memory usage:

root@ss03:~# free
             total       used       free     shared    buffers     cached
Mem:      16463396    6700900    9762496          0    2805756     391884
-/+ buffers/cache:    3503260   12960136
Swap:      8319948          0    8319948


Still I didn't update the kernel, so I am using KVM-88 with kernel
2.6.24-19. I would like to know if this can have some disadvantage, since
in the official site of the project I was reading that kernel has to be
2.6.25 or newer to run the kvm 76 userspace (or any newer release).

The reason by which I didn't update kernel is because I was with some
problems with the network interface on the host machine. Perhaps this
problem is off-topic, but if somebody can help me with this, then it would
contribute whereupon it can improve the KVM performance.

I've downloaded to the source code of 2.6.30.3 and I compiled it with:

# fakeroot make-kpkg clean
# fakeroot make-kpkg --initrd --append-to-version=-dgb kernel_image kernel_headers

Although the compilation finishes without problems, I am having a
problem booting the operating system. The interfaces fail to load. This
equipment is a Proliant DL380 G5 with two NetXtreme II BCM5708 Ethernet
Gigabit interfaces.

With kernel of the Ubuntu Hardy repositories I did not have any kind of
problem, but with kernel compiled by myself I obtain the following error
message during the boot that it suggests cannot accede to firmware of
this network interface:

Aug  4 17:38:39 ss03 kernel: [    4.750023] bnx2 0000:03:00.0: firmware: requesting bnx2/bnx2-mips-06-4.6.16.fw
Aug  4 17:38:39 ss03 kernel: [    4.751070] bnx2: Can't load firmware file "bnx2/bnx2-mips-06-4.6.16.fw"
Aug  4 17:38:39 ss03 kernel: [    4.751141] bnx2 0000:03:00.0: PCI INT A disabled
Aug  4 17:38:39 ss03 kernel: [    4.751167] bnx2: probe of 0000:03:00.0 failed with error -2
Aug  4 17:38:39 ss03 kernel: [    4.751244] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

The peculiar thing is that if I do:

# rmmod bnx2
# modprobe bnx2

Yes it loads the interfaces. I read something of this in a bug report of
Debian GNU/Linux [1] and tried with the suggestion to do:

# update-initramfs -u

The files seem to be:

# find / | grep bnx
/boot/lib/modules/2.6.30.3-dgb/kernel/drivers/net/bnx2.ko
/sys/bus/pci/drivers/bnx2
/sys/bus/pci/drivers/bnx2/0000:03:00.0
/sys/bus/pci/drivers/bnx2/0000:05:00.0
/sys/bus/pci/drivers/bnx2/module
/sys/bus/pci/drivers/bnx2/uevent
/sys/bus/pci/drivers/bnx2/unbind
/sys/bus/pci/drivers/bnx2/bind
/sys/bus/pci/drivers/bnx2/new_id
/sys/bus/pci/drivers/bnx2/remove_id
/sys/module/bnx2
/sys/module/bnx2/holders
/sys/module/bnx2/version
/sys/module/bnx2/srcversion
/sys/module/bnx2/initstate
/sys/module/bnx2/refcnt
/sys/module/bnx2/sections
/sys/module/bnx2/sections/.note.gnu.build-id
/sys/module/bnx2/sections/.text
/sys/module/bnx2/sections/.exit.text
/sys/module/bnx2/sections/.devinit.text
/sys/module/bnx2/sections/.init.text
/sys/module/bnx2/sections/.devexit.text
/sys/module/bnx2/sections/.rodata
/sys/module/bnx2/sections/.rodata.str1.1
/sys/module/bnx2/sections/.smp_locks
/sys/module/bnx2/sections/__bug_table
/sys/module/bnx2/sections/.rodata.str1.8
/sys/module/bnx2/sections/__param
/sys/module/bnx2/sections/.devinit.rodata
/sys/module/bnx2/sections/.data
/sys/module/bnx2/sections/.devinit.data
/sys/module/bnx2/sections/.gnu.linkonce.this_module
/sys/module/bnx2/sections/.bss
/sys/module/bnx2/sections/.symtab
/sys/module/bnx2/sections/.strtab
/sys/module/bnx2/notes
/sys/module/bnx2/notes/.note.gnu.build-id
/sys/module/bnx2/drivers
/sys/module/bnx2/drivers/pci:bnx2
/lib/firmware/bnx2
/lib/firmware/bnx2/bnx2-rv2p-06-4.6.16.fw
/lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw
/lib/firmware/bnx2/bnx2-rv2p-09-4.6.15.fw
/lib/firmware/bnx2/bnx2-mips-09-4.6.17.fw
/lib/modules/2.6.30.4-dgb/kernel/drivers/net/bnx2.ko
/lib/modules/2.6.24-19-server/kernel/drivers/net/bnx2.ko


But after boot I have again the same problem.

/etc/initramfs-tools/modules doesn't make reference to some module. I do
not see that in Ubuntu Hardy firmware-bnx2 is a package, although am not
sure either if it is necessary with kernel compiled from the sources.

With 2.6.30.4 it happens the same exactly.


Thanks for your reply.

Regards,
Daniel

[1] http://www.linux-kvm.org/page/Downloads
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Very high memory usage with KVM

[Avi Kivity]

On 08/08/2009 03:54 AM, Daniel Bareiro wrote:
>> My guess is that it is due to poor swapping with pre-2.6.27 hosts.
>> 15 GB used out of 16GB total is just 6% reserve, which may be a bit
>> too low.  With a 2.6.27 host some small amount of memory would be
>> swapped out, before that you'd see thrashing.  Another way to check
>> is to drop one guest (or reduce total memory needed by 1GB) and see
>> if you get the same results or if things improve.
>>      
>
> I was testing in another equipment of similar characteristics with 4 VMs of
> 3.5 GiB of RAM on LVM and rather less the 16 GB of RAM in use in the host
> machine and 4 GIB available of the 8 GIB of swap, and after to update to
> KVM-88 the use of virtual memory improved enough.
>
> At the moment these are the statistics of memory usage:
>
> root@ss03:~# free
>               total       used       free     shared    buffers     cached
> Mem:      16463396    6700900    9762496          0    2805756     391884
> -/+ buffers/cache:    3503260   12960136
> Swap:      8319948          0    8319948
>
>
> Still I didn't update the kernel, so I am using KVM-88 with kernel
> 2.6.24-19. I would like to know if this can have some disadvantage, since
> in the official site of the project I was reading that kernel has to be
> 2.6.25 or newer to run the kvm 76 userspace (or any newer release).
>    

I think it's just luck; with kernel < 2.6.27 swapping behaviour will not 
be very good.

> The reason by which I didn't update kernel is because I was with some
> problems with the network interface on the host machine. Perhaps this
> problem is off-topic, but if somebody can help me with this, then it would
> contribute whereupon it can improve the KVM performance.
>
>    

Please take it to netdev, perhaps they can help you with this.

-- 
error compiling committee.c: too many arguments to function

Re: Very high memory usage with KVM

[Bernhard Held]

Hi Daniel!

> Aug  4 17:38:39 ss03 kernel: [    4.750023] bnx2 0000:03:00.0: firmware: requesting bnx2/bnx2-mips-06-4.6.16.fw
> Aug  4 17:38:39 ss03 kernel: [    4.751070] bnx2: Can't load firmware file "bnx2/bnx2-mips-06-4.6.16.fw"
> Aug  4 17:38:39 ss03 kernel: [    4.751141] bnx2 0000:03:00.0: PCI INT A disabled
> Aug  4 17:38:39 ss03 kernel: [    4.751167] bnx2: probe of 0000:03:00.0 failed with error -2
> Aug  4 17:38:39 ss03 kernel: [    4.751244] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Is bnx2.ko loaded by initramfs? Then the initramfs must contain the requested 
firmware file.

Another possibility is to set FIRMWARE_IN_KERNEL in the kernel config.

Bernhard

Re: Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Monday, 10 August 2009 08:40:48 +0200,
Bernhard Held wrote:

> Hi Daniel!

Hi Bernhard!
 
> >Aug  4 17:38:39 ss03 kernel: [    4.750023] bnx2 0000:03:00.0: firmware: requesting bnx2/bnx2-mips-06-4.6.16.fw
> >Aug  4 17:38:39 ss03 kernel: [    4.751070] bnx2: Can't load firmware file "bnx2/bnx2-mips-06-4.6.16.fw"
> >Aug  4 17:38:39 ss03 kernel: [    4.751141] bnx2 0000:03:00.0: PCI INT A disabled
> >Aug  4 17:38:39 ss03 kernel: [    4.751167] bnx2: probe of 0000:03:00.0 failed with error -2
> >Aug  4 17:38:39 ss03 kernel: [    4.751244] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

> Is bnx2.ko loaded by initramfs? Then the initramfs must contain the
> requested firmware file.
> 
> Another possibility is to set FIRMWARE_IN_KERNEL in the kernel config.

I've set in the kernel config:


CONFIG_FIRMWARE_IN_KERNEL=y


Although I don't see the asked file in initramfs. The file with full
path is /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw:

# zcat /boot/initrd.img-2.6.30.4-dgb | cpio -t | grep bnx2
lib/modules/2.6.30.4-dgb/kernel/drivers/net/bnx2.ko
300859 blocks

Whereupon this variable is set to 'y' would not have to be sufficient?

<just_guessing>
1) Perhaps firmware must be in some other directory.
2) Perhaps there is no access to the FS in /lib at the moment that loads
drivers.
</just_guessing>


Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Very high memory usage with KVM

[Bernhard Held]

Hi Daniel!

 > I've set in the kernel config:
 >
 > CONFIG_FIRMWARE_IN_KERNEL=y
And what happened? Are you sure you recompiled and booted the new kernel with 
CONFIG_FIRMWARE_IN_KERNEL=y?

 > Although I don't see the asked file in initramfs. The file with full
 > path is /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw:
 >
 > # zcat /boot/initrd.img-2.6.30.4-dgb | cpio -t | grep bnx2
 > lib/modules/2.6.30.4-dgb/kernel/drivers/net/bnx2.ko
 > 300859 blocks
 >
 > Whereupon this variable is set to 'y' would not have to be sufficient?
 >
 > <just_guessing>
 > 1) Perhaps firmware must be in some other directory.
 > 2) Perhaps there is no access to the FS in /lib at the moment that loads
 > drivers.
 > </just_guessing>
Obviously bnx2.ko is loaded from initramfs. At that time the root filesystem is 
not accessible. I expected CONFIG_FIRMWARE_IN_KERNEL=y is sufficient to solve 
the problem.

Here is what I would try:
- make sure the firmware is included in your kernel, or
- keep bnx2.ko from including in initramfs (unless you boot via network), or
- include /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw in initramfs

HTH
Bernhard

Re: Very high memory usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 2338 bytes --]

On Monday, 10 August 2009 18:15:17 +0200,
Bernhard Held wrote:

> Hi Daniel!

Hi, Bernhard!
 
> > I've set in the kernel config:
> >
> > CONFIG_FIRMWARE_IN_KERNEL=y

> And what happened? Are you sure you recompiled and booted the new
> kernel with CONFIG_FIRMWARE_IN_KERNEL=y?
>
> [...]
> 
> Here is what I would try:
> - make sure the firmware is included in your kernel, or
> - keep bnx2.ko from including in initramfs (unless you boot via network), or
> - include /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw in initramfs

According to the tests that I was doing, the CONFIG_FIRMWARE_IN_KERNEL=y
option is not sufficient. Also it is necessary to use CONFIG_BNX2=y.

According to which I was reading, with the CONFIG_FIRMWARE_IN_KERNEL=y
option, the driver is including in kernel as long as driver it is
compiled in kernel and not in modular form. In order to obtain this it
is necessary to use the CONFIG_BNX2=y option. This option is established
following the path Device Drivers -> Network device support -> Ethernet
(1000 Mbit) of the ncurses menu. Then, we marked with asterisk Broadcom
NetXtremeII support here.

With this, the interfaces were configured during bootstrapping without
problems:

[    2.805440] Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.1 (May 6, 2009)
[    2.805458] bnx2 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    3.410016] bnx2 0000:03:00.0: firmware: using built-in firmware bnx2/bnx2-mips-06-4.6.16.fw
[    3.410019] bnx2 0000:03:00.0: firmware: using built-in firmware bnx2/bnx2-rv2p-06-4.6.16.fw
[    3.410139] eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 16, node addr 00:21:5a:d5:dc:42
[    3.410190] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    4.020012] bnx2 0000:05:00.0: firmware: using built-in firmware bnx2/bnx2-mips-06-4.6.16.fw
[    4.020014] bnx2 0000:05:00.0: firmware: using built-in firmware bnx2/bnx2-rv2p-06-4.6.16.fw
[    4.020113] eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem fa000000, IRQ 17, node addr 00:21:5a:d5:dc:40


Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]