Re: [KVM-AUTOTEST PATCH 1/2] Add KSM test

[Dor Laor <dlaor@redhat.com>]

On 09/29/2009 05:50 PM, Lucas Meneghel Rodrigues wrote:
> On Fri, 2009-09-25 at 05:22 -0400, Jiri Zupka wrote:
>> ----- "Dor Laor"<dlaor@redhat.com>  wrote:
>>
>>> On 09/16/2009 04:09 PM, Jiri Zupka wrote:
>>>>
>>>> ----- "Dor Laor"<dlaor@redhat.com>   wrote:
>>>>
>>>>> On 09/15/2009 09:58 PM, Jiri Zupka wrote:
>>>>>>> After a quick review I have the following questions:
>>>>>>> 1. Why did you implement the guest tool in 'c' and not in
>>> python?
>>>>>>>      Python is much simpler and you can share some code with the
>>>>> server.
>>>>>>>      This 'test protocol' would also be easier to understand this
>>>>> way.
>>>>>>
>>>>>> We need speed and the precise control of allocate memory in
>>> pages.
>>>>>>
>>>>>>> 2. IMHO there is no need to use select, you can do blocking
>>> read.
>>>>>>
>>>>>> We replace socket communication by interactive program
>>> communication
>>>>> via ssh/telnet
>>>>>>
>>>>>>> 3. Also you can use plain malloc without the more complex ( a
>>> bit)
>>>>> mmap.
>>>>>>
>>>>>> We need address exactly the memory pages. We can't allow shift of
>>>>> the data in memory.
>>>>>
>>>>> You can use the tmpfs+dd idea instead of the specific program as I
>>>>> detailed before. Maybe some other binary can be used. My intention
>>> is
>>>>> to
>>>>> simplify the test/environment as much as possible.
>>>>>
>>>>
>>>> We need compatibility with others system, like Windows etc..
>>>> We want to add support for others system in next version
>>>
>>> KSM is a host feature and should be agnostic to the guest.
>>> Also I don't think your code will compile on windows...
>>
>> Yes, I think you have true.
>
> First of all, sorry, I am doing the best I can to review carefully all
> the patch queue, and as KSM is a more involved feature that I am not
> very familiar with, I need a bit more time to review it!
>
>> But because we need generate special data to pages in memory.
>> We need use script on guest side of test. Because communication
>> over ssh is to slow to transfer lot of GB of special data to guests.
>>
>> We can use optimized C program which is 10x and more faster than
>> python script on native system. Heavy load of virtual guest can
>> make some performance problem.
>
> About code compiling under windows, I guess making a native windows c or
> c++ program is an option, I generally agree with your reasoning, this
> case seems to be better covered with a c program. Will get into it in
> more detail ASAP...
>
>> We can use tmpfs but with python script to generate special data.
>> We can't use dd with random because we need test some special case.
>> (change only last 96B of page etc.. )
>>
>>
>> What do you think about it?


I think it can be done with some simple scripting and it will be fast 
enough and more importantly, easier to understand and to change in the 
future.

Here is a short example for creating lots of identical pages that 
contain '0' apart for the last two bytes. If you'll run it in a single 
guest you should expect to save lots of memory. Then you can change the 
last bytes to random value and see the memory consumption grow:
[Remember to cancel the guest swap to keep it in the guest ram]

dd if=/dev/zero of=template  count=1 bs=4094
echo '1' >> template
cp template large_file
for ((i=0;i<10;i++)) do dd if=large_file of=large_file conv=notrunc 
oflag=append > /dev/null 2>&1 ; done

It creates a 4k*2^10 file with identical pages (since it's on tmpfs with 
no swap)

Can you try it? It should be far simpler than the original option.

Thanks,
Dor