Swap usage with KVM

Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 5390 bytes --]

Hi all!

I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
0.12.3 compiled with the source code obtained from the official site of
KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All
this is installed on an HP Proliant DL380 G6 with two Xeon E5530
quadcore processors and 16 GiB of RAM which has two VMs with the
following configuration of memory:

Hostname       |      RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL          |   14 GiB

Initially the host was created with a swap partition of 1 GiB, but today
we found that the use of swap quickly began to grow increasingly.
Therefore, as a contingency, we had to hot-add a logical volume of 1 GB
of swap on the VMHost. Is 'normal' this use of memory?

I copy the Nagios Service Log Entries for the VMHost:

Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
100% free (956 MB out of 956 MB)
09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
99% free (939 MB out of 956 MB)
10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
79% free (754 MB out of 956 MB)
11-07-2010 00:00:00     11-07-2010 07:08:17     0d 7h 8m 17s    SERVICE OK (HARD)       SWAP OK -
51% free (482 MB out of 956 MB)
11-07-2010 07:08:17     11-07-2010 10:41:07     0d 3h 32m 50s   SERVICE WARNING (HARD)  SWAP WARNING
- 29% free (272 MB out of 956 MB)
11-07-2010 10:41:07     11-07-2010 10:45:57     0d 0h 4m 50s    SERVICE CRITICAL (HARD) SWAP
CRITICAL - 9% free (83 MB out of 956 MB)
---------------------------------------------------------------------------------------------------

I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical
volume that maintains the logical volumes that are used for VM's disks.

The Nagios Service Log Entries for VMs shows no excessive use of swap in
the window of time when the problem occurred in the VMHost:

Aps4:

Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
100% free (2850 MB out of 2863 MB)
07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
98% free (2797 MB out of 2863 MB)
08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
99% free (2812 MB out of 2863 MB)
09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
98% free (2784 MB out of 2863 MB)
10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
97% free (2754 MB out of 2863 MB)
11-07-2010 00:00:00     11-07-2010 11:53:38     0d 11h 53m 38s+ SERVICE OK (HARD)       SWAP OK -
100% free (2839 MB out of 2863 MB)
---------------------------------------------------------------------------------------------------

Leela:

Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
97% free (919 MB out of 956 MB)
07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 17:01:36     08-07-2010 17:04:16     0d 0h 2m 40s    SERVICE CRITICAL (HARD) Connection
refused by host
08-07-2010 17:04:16     09-07-2010 00:00:00     0d 6h 55m 44s   SERVICE OK (HARD)       SWAP OK -
97% free (921 MB out of 956 MB)
09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
97% free (921 MB out of 956 MB)
10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
97% free (921 MB out of 956 MB)
11-07-2010 00:00:00     11-07-2010 11:58:29     0d 11h 58m 29s+ SERVICE OK (HARD)       SWAP OK -
97% free (921 MB out of 956 MB)
---------------------------------------------------------------------------------------------------



Unfortunately I could not take much more data because we had to act
quickly, but if you need any additional information, please feel free to
ask.

Thanks in advance for your replies.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 5820 bytes --]

On Sunday, 11 July 2010 12:12:57 -0300,
Daniel Bareiro wrote:

> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
> 0.12.3 compiled with the source code obtained from the official site
> of KVM and Linux 2.6.32.12 compiled from source code of kernel.org.
> All this is installed on an HP Proliant DL380 G6 with two Xeon E5530
> quadcore processors and 16 GiB of RAM which has two VMs with the
> following configuration of memory:
> 
> Hostname       |      RAM
> ===============+===============
> Aps4           |    7 GiB
> Leela          |    7 GiB
> ===============+===============
> TOTAL          |   14 GiB
> 
> Initially the host was created with a swap partition of 1 GiB, but
> today we found that the use of swap quickly began to grow
> increasingly.  Therefore, as a contingency, we had to hot-add a
> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of
> memory?
> 
> I copy the Nagios Service Log Entries for the VMHost:
> 
> Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 100% free (956 MB out of 956 MB)
> 07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 100% free (956 MB out of 956 MB)
> 08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
> 100% free (956 MB out of 956 MB)
> 09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 99% free (939 MB out of 956 MB)
> 10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 79% free (754 MB out of 956 MB)
> 11-07-2010 00:00:00     11-07-2010 07:08:17     0d 7h 8m 17s    SERVICE OK (HARD)       SWAP OK -
> 51% free (482 MB out of 956 MB)
> 11-07-2010 07:08:17     11-07-2010 10:41:07     0d 3h 32m 50s   SERVICE WARNING (HARD)  SWAP WARNING
> - 29% free (272 MB out of 956 MB)
> 11-07-2010 10:41:07     11-07-2010 10:45:57     0d 0h 4m 50s    SERVICE CRITICAL (HARD) SWAP
> CRITICAL - 9% free (83 MB out of 956 MB)
> ---------------------------------------------------------------------------------------------------
> 
> I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a
> physical volume that maintains the logical volumes that are used for
> VM's disks.
> 
> The Nagios Service Log Entries for VMs shows no excessive use of swap
> in the window of time when the problem occurred in the VMHost:
> 
> Aps4:
> 
> Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 100% free (2850 MB out of 2863 MB)
> 07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 98% free (2797 MB out of 2863 MB)
> 08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
> 99% free (2812 MB out of 2863 MB)
> 09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 98% free (2784 MB out of 2863 MB)
> 10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 97% free (2754 MB out of 2863 MB)
> 11-07-2010 00:00:00     11-07-2010 11:53:38     0d 11h 53m 38s+ SERVICE OK (HARD)       SWAP OK -
> 100% free (2839 MB out of 2863 MB)
> ---------------------------------------------------------------------------------------------------
> 
> Leela:
> 
> Event Start Time        Event End Time  Event Duration  Event/State Type        Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00     07-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 97% free (919 MB out of 956 MB)
> 07-07-2010 00:00:00     08-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 97% free (920 MB out of 956 MB)
> 08-07-2010 00:00:00     08-07-2010 16:41:43     0d 16h 41m 43s  SERVICE OK (HARD)       SWAP OK -
> 97% free (920 MB out of 956 MB)
> 08-07-2010 17:01:36     08-07-2010 17:04:16     0d 0h 2m 40s    SERVICE CRITICAL (HARD) Connection
> refused by host
> 08-07-2010 17:04:16     09-07-2010 00:00:00     0d 6h 55m 44s   SERVICE OK (HARD)       SWAP OK -
> 97% free (921 MB out of 956 MB)
> 09-07-2010 00:00:00     10-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 97% free (921 MB out of 956 MB)
> 10-07-2010 00:00:00     11-07-2010 00:00:00     1d 0h 0m 0s     SERVICE OK (HARD)       SWAP OK -
> 97% free (921 MB out of 956 MB)
> 11-07-2010 00:00:00     11-07-2010 11:58:29     0d 11h 58m 29s+ SERVICE OK (HARD)       SWAP OK -
> 97% free (921 MB out of 956 MB)
> ---------------------------------------------------------------------------------------------------
> 
> 
> 
> Unfortunately I could not take much more data because we had to act
> quickly, but if you need any additional information, please feel free to
> ask.

Has anyone experienced something like this? Avi? I remember late last
year there was a regression in Linux swapping and Rik and Hugh were
working on it. Are you aware of any?

Thanks in advance for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[Rik van Riel]

On 07/11/2010 03:12 PM, Daniel Bareiro wrote:
> On Sunday, 11 July 2010 12:12:57 -0300,
> Daniel Bareiro wrote:
>
>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
>> 0.12.3 compiled with the source code obtained from the official site
>> of KVM and Linux 2.6.32.12 compiled from source code of kernel.org.
>> All this is installed on an HP Proliant DL380 G6 with two Xeon E5530
>> quadcore processors and 16 GiB of RAM which has two VMs with the
>> following configuration of memory:
>>
>> Hostname       |      RAM
>> ===============+===============
>> Aps4           |    7 GiB
>> Leela          |    7 GiB
>> ===============+===============
>> TOTAL          |   14 GiB
>>
>> Initially the host was created with a swap partition of 1 GiB, but
>> today we found that the use of swap quickly began to grow
>> increasingly.  Therefore, as a contingency, we had to hot-add a
>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of
>> memory?

That depends on what is going on in the host.

Did you notice any performance issues in the guest when
you started using swap?

> Has anyone experienced something like this? Avi? I remember late last
> year there was a regression in Linux swapping and Rik and Hugh were
> working on it. Are you aware of any?

That one should have been fixed.

-- 
All rights reversed

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 2236 bytes --]

Hi, Rik.

On Sunday, 11 July 2010 17:49:43 -0400,
Rik van Riel wrote:

>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>> of kernel.org. All this is installed on an HP Proliant DL380 G6 with
>>> two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
>>> VMs with the following configuration of memory:
>>>
>>> Hostname       |      RAM
>>> ===============+===============
>>> Aps4           |    7 GiB
>>> Leela          |    7 GiB
>>> ===============+===============
>>> TOTAL          |   14 GiB
>>>
>>> Initially the host was created with a swap partition of 1 GiB, but
>>> today we found that the use of swap quickly began to grow
>>> increasingly.  Therefore, as a contingency, we had to hot-add a
>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>> of memory?

> That depends on what is going on in the host.
>
> Did you notice any performance issues in the guest when you started
> using swap?

After the logical volume of 1 GB that I added when I found this problem
(being the operating system with 2 GB), I added other 3 GB to have a
little more margin, but today I got again a new alert of Nagios:

        
Swap usage     WARNING    [...]    SWAP WARNING - 30% free   (1490 MB out of 5052 MB)


I don't see performance issue in the VMs.

Marcelo Tosatti recommended me to apply the EPT patch
(6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
production? Are there plans that this patch is applied on some version
of Linux stable?

Also it can be advisable to update to qemu-kvm 0.12.4 considering what
David Weber said:

http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599


Thanks for your reply.

Regards,
Daniel
-- 
Mi frase del día:
Collaboration, n.:
	A literary partnership based on the false assumption that the
	other fellow can spell.

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
15:28:17 up 23 days, 20:20, 12 users,  load average: 0.16, 0.14, 0.09

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[Rik van Riel]

On 08/02/2010 02:57 PM, Daniel Bareiro wrote:
> Hi, Rik.
>
> On Sunday, 11 July 2010 17:49:43 -0400,
> Rik van Riel wrote:
>
>>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>>> of kernel.org. All this is installed on an HP Proliant DL380 G6 with
>>>> two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
>>>> VMs with the following configuration of memory:
>>>>
>>>> Hostname       |      RAM
>>>> ===============+===============
>>>> Aps4           |    7 GiB
>>>> Leela          |    7 GiB
>>>> ===============+===============
>>>> TOTAL          |   14 GiB
>>>>
>>>> Initially the host was created with a swap partition of 1 GiB, but
>>>> today we found that the use of swap quickly began to grow
>>>> increasingly.  Therefore, as a contingency, we had to hot-add a
>>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>>> of memory?
>
>> That depends on what is going on in the host.
>>
>> Did you notice any performance issues in the guest when you started
>> using swap?
>
> After the logical volume of 1 GB that I added when I found this problem
> (being the operating system with 2 GB), I added other 3 GB to have a
> little more margin, but today I got again a new alert of Nagios:
>
>
> Swap usage     WARNING    [...]    SWAP WARNING - 30% free   (1490 MB out of 5052 MB)
>
>
> I don't see performance issue in the VMs.
>
> Marcelo Tosatti recommended me to apply the EPT patch
> (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
> production? Are there plans that this patch is applied on some version
> of Linux stable?

Yes, that patch is safe and definitely recommended.

-- 
All rights reversed

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 2263 bytes --]

Hi, Rik.

On Monday, 02 August 2010 15:31:47 -0400,
Rik van Riel wrote:

>>>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>>>> of kernel.org. All this is installed on an HP Proliant DL380 G6
>>>>> with two Xeon E5530 quadcore processors and 16 GiB of RAM which
>>>>> has two VMs with the following configuration of memory:
>>>>>
>>>>> Hostname       |      RAM
>>>>> ===============+===============
>>>>> Aps4           |    7 GiB
>>>>> Leela          |    7 GiB
>>>>> ===============+===============
>>>>> TOTAL          |   14 GiB
>>>>>
>>>>> Initially the host was created with a swap partition of 1 GiB, but
>>>>> today we found that the use of swap quickly began to grow
>>>>> increasingly.  Therefore, as a contingency, we had to hot-add a
>>>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>>>> of memory?

>>> That depends on what is going on in the host.
>>>
>>> Did you notice any performance issues in the guest when you started
>>> using swap?

>> After the logical volume of 1 GB that I added when I found this
>> problem (being the operating system with 2 GB), I added other 3 GB to
>> have a little more margin, but today I got again a new alert of
>> Nagios:
>>
>>
>> Swap usage     WARNING    [...]    SWAP WARNING - 30% free   (1490 MB out of 5052 MB)
>>
>>
>> I don't see performance issue in the VMs.
>>
>> Marcelo Tosatti recommended me to apply the EPT patch
>> (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
>> production? Are there plans that this patch is applied on some version
>> of Linux stable?

> Yes, that patch is safe and definitely recommended.

Perfect. Thanks for so fast answer.

And there are some estimates of when this patch is in Linux stable?

Regards,
Daniel
-- 
Mi frase del día:
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
16:44:53 up 23 days, 21:37, 11 users,  load average: 0.02, 0.06, 0.08

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[Rik van Riel]

On 08/02/2010 03:52 PM, Daniel Bareiro wrote:

> And there are some estimates of when this patch is in Linux stable?

It should be there already in 2.6.33-stable and 2.6.34-stable.

-- 
All rights reversed

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

El lunes 02 de agosto del 2010 a las 16:22:14,
Rik van Riel escribió:

>> And there are some estimates of when this patch is in Linux stable?

> It should be there already in 2.6.33-stable and 2.6.34-stable.

Great! Thanks for your reply.

Regards,
Daniel
-- 
Mi frase del día:
i dont even know if it makes sense at all :) This is an experimental patch
for an experimental kernel :))
		-- Ingo Molnar on linux-kernel

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
18:38:51 up 23 days, 23:31, 11 users,  load average: 0.17, 0.16, 0.10

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM (and KSM)

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

Hi, Rik.

On Monday, 02 August 2010 16:22:14 -0400,
Rik van Riel wrote:

>> And there are some estimates of when this patch is in Linux stable?

> It should be there already in 2.6.33-stable and 2.6.34-stable.

In a test environment I have Linux 2.6.34.5 compiled with the source
code from kernel.org and qemu-kvm 0.12.4 installed from the Backports
repository of Debian GNU/Linux.

In this environment I have running the following virtual machines:

* 2 x OpenBSD (2 x 512 MB)            |
* 2 x Debian GNU/Linux (2 x 256 MB)   |
* 2 x Debian GNU/Linux (2 x 512 MB)   +--= 11008 MB
* 1 x Centos 5.5 (1 x 256 MB)         |
* 8 x Debian GNU/Linux (8 x 1024 MB)  |

I'm just starting to test KSM, so I was using as reference the read of
the tests made by RedHat about run as many as 52 Windows XP VMs with 1
GB of RAM each on a server with just 16 GB of RAM.

In the previous case the ratio would be 52/16 = 3.25. In my case the
VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH
tests do not talk about the amount of swap used in that case, so I
wonder if a distribution of VMs as I have, it is normal to use 290 MB of
swap.


Regards,
Daniel
-- 
Mi frase del día:
BOFH excuse #335:

the AA battery in the wallclock sends magnetic interference


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
05:57:46 up 48 days, 10:50, 11 users,  load average: 0.01, 0.04, 0.06

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM (and KSM)

[Rik van Riel]

On 08/27/2010 06:04 AM, Daniel Bareiro wrote:

> In the previous case the ratio would be 52/16 = 3.25. In my case the
> VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH
> tests do not talk about the amount of swap used in that case, so I
> wonder if a distribution of VMs as I have, it is normal to use 290 MB of
> swap.

Given that you have 10.75GB worth of virtual machine on a
system with 4GB of RAM, I'd say that 290MB of swap is not
out of the ordinary.

-- 
All rights reversed

Re: Swap usage with KVM

[Freddie Cash]

On Sun, Jul 11, 2010 at 8:12 AM, Daniel Bareiro <daniel-listas@gmx.net> wrote:
> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
> 0.12.3 compiled with the source code obtained from the official site of
> KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All
> this is installed on an HP Proliant DL380 G6 with two Xeon E5530
> quadcore processors and 16 GiB of RAM which has two VMs with the
> following configuration of memory:

Are you using virtio drivers in the VMs?

There was an issue with KVM-72 and virtio that leaks memory in the
host until all RAM and swap is used (inside the VMs, no swap is used).
 It was supposed to be fixed in KVM-80-something, though.

Perhaps something similar is happening again?  If you switch the disks
to scsi instead of virtio, does the problem go away?

We are running KVM-72 on Debian 5.0 and have run into this issue.
We'll be upgrading our hosts this month to fix this.

-- 
Freddie Cash
fjwcash@gmail.com

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]

Hi, Freddie.

On Sunday, 11 July 2010 14:05:38 -0700,
Freddie Cash wrote:

> > I have an installation with Debian GNU/Linux 5.0.4 amd64 with
> > qemu-kvm 0.12.3 compiled with the source code obtained from the
> > official site of KVM and Linux 2.6.32.12 compiled from source code
> > of kernel.org. All this is installed on an HP Proliant DL380 G6 with
> > two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
> > VMs with the following configuration of memory:

> Are you using virtio drivers in the VMs?
> 
> There was an issue with KVM-72 and virtio that leaks memory in the
> host until all RAM and swap is used (inside the VMs, no swap is used).
> It was supposed to be fixed in KVM-80-something, though.
> 
> Perhaps something similar is happening again?  If you switch the disks
> to scsi instead of virtio, does the problem go away?
> 
> We are running KVM-72 on Debian 5.0 and have run into this issue.
> We'll be upgrading our hosts this month to fix this.

Yes, we are using Virtio drivers for networking and storage in both VMs
with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
Lenny Backports repositories. For VMHost, we are using a stable version
of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
qemu-kvm 0.12.3 compiled with the source code obtained from the official
site of KVM.

This is the syntax I'm using to boot the virtual machines:


 8587 ?        Sl   6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio
-drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net
nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor
telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait

 9769 ?        Rl   11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio
-drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive
file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio
-drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive
file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio
-drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive
file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net
nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor
telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait


To make the switch from Virtio to SCSI I would have to shut down the
hosts, which would not be a good idea whereas are two productive
systems. At least, before doing so I would be sure of what might be the
problem.

Taking a current measurement in VMHost with free, I got the following:


ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16406504      55084          0       2920      21504
-/+ buffers/cache:   16382080      79508
Swap:      2028492     983140    1045352


It draws attention to me that thinking about initially leaving a margin
of 2 GB of RAM for the VMHost, already it has used almost half of swap.


Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1: Type: text/plain, Size: 4755 bytes --]

On Sunday, 11 July 2010 19:08:58 -0300,
Daniel Bareiro wrote:

> > > I have an installation with Debian GNU/Linux 5.0.4 amd64 with
> > > qemu-kvm 0.12.3 compiled with the source code obtained from the
> > > official site of KVM and Linux 2.6.32.12 compiled from source code
> > > of kernel.org. All this is installed on an HP Proliant DL380 G6
> > > with two Xeon E5530 quadcore processors and 16 GiB of RAM which
> > > has two VMs with the following configuration of memory:

> > Are you using virtio drivers in the VMs?
> > 
> > There was an issue with KVM-72 and virtio that leaks memory in the
> > host until all RAM and swap is used (inside the VMs, no swap is
> > used). It was supposed to be fixed in KVM-80-something, though.
> > 
> > Perhaps something similar is happening again?  If you switch the
> > disks to scsi instead of virtio, does the problem go away?
> > 
> > We are running KVM-72 on Debian 5.0 and have run into this issue.
> > We'll be upgrading our hosts this month to fix this.

> Yes, we are using Virtio drivers for networking and storage in both
> VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64
> from Lenny Backports repositories. For VMHost, we are using a stable
> version of KVM with Linux 2.6.32.12 compiled from source code of
> kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained
> from the official site of KVM.
> 
> This is the syntax I'm using to boot the virtual machines:
> 
> 
>  8587 ?        Sl   6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
> file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive
> file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio
> -drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net
> nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor
> telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait
> 
>  9769 ?        Rl   11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
> file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive
> file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio
> -drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive
> file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio
> -drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive
> file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio
> -drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive
> file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net
> nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor
> telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait

> To make the switch from Virtio to SCSI I would have to shut down the
> hosts, which would not be a good idea whereas are two productive
> systems. At least, before doing so I would be sure of what might be
> the problem.
> 
> Taking a current measurement in VMHost with free, I got the following:
> 
> 
> ss04:~# free
>              total       used       free     shared    buffers     cached
> Mem:      16461588   16406504      55084          0       2920      21504
> -/+ buffers/cache:   16382080      79508
> Swap:      2028492     983140    1045352
> 
> 
> It draws attention to me that thinking about initially leaving a margin
> of 2 GB of RAM for the VMHost, already it has used almost half of swap.

This is a current measurement I've taken in both the VMs and in VMHost:

* VMHost:


ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16405140      56448          0       3496      18604
-/+ buffers/cache:   16383040      78548
Swap:      5174220    2401552    2772668


* Aps4:

aps4:~# free
             total       used       free     shared    buffers     cached
Mem:       7164300    7120192      44108          0      23108     239076
-/+ buffers/cache:    6858008     306292
Swap:      2931820      14084    2917736


* Leela:

leela:~# free
             total       used       free     shared    buffers     cached
Mem:       7163836    6905224     258612          0     123380    6282816
-/+ buffers/cache:     499028    6664808
Swap:       979924      35640     944284


As you can see, I added more swap in VMHost for more margin, but
currently only 54% is free.



Thanks in advance for your replies.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Swap usage with KVM

[David Weber]

> Yes, we are using Virtio drivers for networking and storage in both VMs
> with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
> Lenny Backports repositories. For VMHost, we are using a stable version
> of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
> qemu-kvm 0.12.3 compiled with the source code obtained from the official
> site of KVM.
> 

Afaik this should be this bug
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599

try upgrading to 0.12.4 or backport this commit
http://git.kernel.org/?p=virt/kvm/qemu-
kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960

David

Re: Swap usage with KVM

[Daniel Bareiro]

[-- Attachment #1.1: Type: text/plain, Size: 1138 bytes --]

Hi, David.

On Tuesday, 20 July 2010 21:18:09 +0200,
David Weber wrote:

> > Yes, we are using Virtio drivers for networking and storage in both VMs
> > with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
> > Lenny Backports repositories. For VMHost, we are using a stable version
> > of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
> > qemu-kvm 0.12.3 compiled with the source code obtained from the official
> > site of KVM.

> Afaik this should be this bug
> http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
> 
> try upgrading to 0.12.4 or backport this commit
> http://git.kernel.org/?p=virt/kvm/qemu-
> kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960

Interesting... I'll try upgrading to qemu-kvm 0.12.4.

Marcelo Tosatti also recommended me to apply this patch I'm attaching.
Someone could confirm whether this is already included in Linux stable?

Thanks for your reply.

Regards,
Daniel
-- 
Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

[-- Attachment #1.2: kvm-vmx-accessed-bit.patch --]
[-- Type: text/x-diff, Size: 1691 bytes --]

commit 6316e1c8c6af6ccb55ff8564231710660608f46c
Author: Rik van Riel <riel@redhat.com>
Date:   Wed Feb 3 16:11:03 2010 -0500

    KVM: VMX: emulate accessed bit for EPT
    
    Currently KVM pretends that pages with EPT mappings never got
    accessed.  This has some side effects in the VM, like swapping
    out actively used guest pages and needlessly breaking up actively
    used hugepages.
    
    We can avoid those very costly side effects by emulating the
    accessed bit for EPT PTEs, which should only be slightly costly
    because pages pass through page_referenced infrequently.
    
    TLB flushing is taken care of by kvm_mmu_notifier_clear_flush_young().
    
    This seems to help prevent KVM guests from being swapped out when
    they should not on my system.
    
    Signed-off-by: Rik van Riel <riel@redhat.com>
    Signed-off-by: Avi Kivity <avi@redhat.com>

diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 913ef4b..b8da671 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -838,9 +838,15 @@ static int kvm_age_rmapp(struct kvm *kvm, unsigned long *rmapp,
 	u64 *spte;
 	int young = 0;
 
-	/* always return old for EPT */
+	/*
+	 * Emulate the accessed bit for EPT, by checking if this page has
+	 * an EPT mapping, and clearing it if it does. On the next access,
+	 * a new EPT mapping will be established.
+	 * This has some overhead, but not as much as the cost of swapping
+	 * out actively used pages or breaking up actively used hugepages.
+	 */
 	if (!shadow_accessed_mask)
-		return 0;
+		return kvm_unmap_rmapp(kvm, rmapp, data);
 
 	spte = rmap_next(kvm, rmapp, NULL);
 	while (spte) {

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]