2.6.23.1-rt4 and kvm 48

2.6.23.1-rt4 and kvm 48

[David Brown]

Uhm, not sure who to send this too...

I thought I'd try out the realtime patch set and it didn't work at all
with kvm. The console didn't dump anything and the system completely
locked up.

Anyone have any suggestions as to how to get more output on this issue?

It got to the point of bringing up the tap interface and attaching it
to the bridge but that was about it for the console messages.

Thanks,
- David Brown

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

David Brown wrote:
> Uhm, not sure who to send this too...
>
> I thought I'd try out the realtime patch set and it didn't work at all
> with kvm. The console didn't dump anything and the system completely
> locked up.
>
> Anyone have any suggestions as to how to get more output on this issue?
>   

Make sure CONFIG_PREEMPT_NOTIFIERS is enabled. Jan Kiszka reported 
success with a similar configuration.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Carsten Emde]

Avi Kivity wrote:
> David Brown wrote:
>> I thought I'd try out the realtime patch set and it didn't work at all
>> with kvm. The console didn't dump anything and the system completely
>> locked up.
Up to now, the unmodified kvm module never worked with any RT kernel. 
This would only change, if RT patched kernels were routinely used in the 
kvm release tests which is, for the time being, not the case.

However, tglx once provided a hand-crafted kvm version that can be used 
together with a selected RT kernel -> see "Latest Stable" 
(http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel.0.html) 
realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.

We used these modules to carry out a number of "proof of concept" tests 
which were very successful. The realtime capabilities of the host system 
remained unchanged irrespective of whether the kvm guest system (no 
graphics) was running or not. This was even the case while the guest 
system was booting.

We are planning to provide another "Latest Stable" realtime kernel 
within the next month or so. It will probably be based on kernel 2.6.23.

	--cbe

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Jan Kiszka]

Carsten Emde wrote:
> Avi Kivity wrote:
>> David Brown wrote:
>>> I thought I'd try out the realtime patch set and it didn't work at all
>>> with kvm. The console didn't dump anything and the system completely
>>> locked up.
> Up to now, the unmodified kvm module never worked with any RT kernel. 
> This would only change, if RT patched kernels were routinely used in the 
> kvm release tests which is, for the time being, not the case.

As far as I understood (but didn't re-test yet), latest kvm release does
include all features required to run over -rt. This was at least the
case for kvm-git I picked up last week and merged into 2.6.23-rt1. The
only issue is that out-of-tree kvm doesn't enable the preemption
notifiers (maybe there were one or two further minor issues, I think to
recall missing smp_ops exports being one, but these were not -rt related).

> 
> However, tglx once provided a hand-crafted kvm version that can be used 
> together with a selected RT kernel -> see "Latest Stable" 
> (http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel.0.html) 
> realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.
> 
> We used these modules to carry out a number of "proof of concept" tests 
> which were very successful. The realtime capabilities of the host system 
> remained unchanged irrespective of whether the kvm guest system (no 
> graphics) was running or not. This was even the case while the guest 
> system was booting.

Interesting result - you've read about the wbinvd issues? Is there no
wbinvd in the bios shipped with older kvm? Which VM extension did you
test, both Intel and AMD? I would bet that your X issues are due to the
same effect. X startup/shutdown involves a lot of wbinvd calls on my
test boxes.

To sum up my findings: kvm over whatever RT kernel is risky
(latency-wise) unless you can audit or even para-virtualise your guest
OS /wrt wbinvd. Maybe we can discuss this also in Linz by the end of the
week. :)

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Alexander Graf]

On Oct 30, 2007, at 10:49 AM, Jan Kiszka wrote:

> Carsten Emde wrote:
>> Avi Kivity wrote:
>>> David Brown wrote:
>>>> I thought I'd try out the realtime patch set and it didn't work  
>>>> at all
>>>> with kvm. The console didn't dump anything and the system  
>>>> completely
>>>> locked up.
>> Up to now, the unmodified kvm module never worked with any RT kernel.
>> This would only change, if RT patched kernels were routinely used  
>> in the
>> kvm release tests which is, for the time being, not the case.
>
> As far as I understood (but didn't re-test yet), latest kvm release  
> does
> include all features required to run over -rt. This was at least the
> case for kvm-git I picked up last week and merged into 2.6.23-rt1. The
> only issue is that out-of-tree kvm doesn't enable the preemption
> notifiers (maybe there were one or two further minor issues, I  
> think to
> recall missing smp_ops exports being one, but these were not -rt  
> related).
>
>>
>> However, tglx once provided a hand-crafted kvm version that can be  
>> used
>> together with a selected RT kernel -> see "Latest Stable"
>> (http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel. 
>> 0.html)
>> realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.
>>
>> We used these modules to carry out a number of "proof of concept"  
>> tests
>> which were very successful. The realtime capabilities of the host  
>> system
>> remained unchanged irrespective of whether the kvm guest system (no
>> graphics) was running or not. This was even the case while the guest
>> system was booting.
>
> Interesting result - you've read about the wbinvd issues? Is there no
> wbinvd in the bios shipped with older kvm? Which VM extension did you
> test, both Intel and AMD? I would bet that your X issues are due to  
> the
> same effect. X startup/shutdown involves a lot of wbinvd calls on my
> test boxes.
>
> To sum up my findings: kvm over whatever RT kernel is risky
> (latency-wise) unless you can audit or even para-virtualise your guest
> OS /wrt wbinvd. Maybe we can discuss this also in Linz by the end  
> of the
> week. :)

As I will be in Linz by the end of the week as well, could you please  
try to do any coordination as to when you meet on the lists (or CC  
me)? I'd like to get the chance to join you on this :-).

Thanks,

Alex

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

Jan Kiszka wrote:
> Interesting result - you've read about the wbinvd issues? Is there no
> wbinvd in the bios shipped with older kvm? Which VM extension did you
> test, both Intel and AMD? I would bet that your X issues are due to the
> same effect. X startup/shutdown involves a lot of wbinvd calls on my
> test boxes.
>
>   

kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
issues there.  I hope that Intel will add similar traps in future 
iterations of VT.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Interesting result - you've read about the wbinvd issues? Is there no
>> wbinvd in the bios shipped with older kvm? Which VM extension did you
>> test, both Intel and AMD? I would bet that your X issues are due to the
>> same effect. X startup/shutdown involves a lot of wbinvd calls on my
>> test boxes.
>>
>>   
> 
> kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
> issues there. 

So kvm can then simply ignore the instructions, or what does it do about
them? I recall some replies in the related thread that some more work
may be required.

> I hope that Intel will add similar traps in future 
> iterations of VT.

Yep.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

Jan Kiszka wrote:
> Avi Kivity wrote:
>   
>> Jan Kiszka wrote:
>>     
>>> Interesting result - you've read about the wbinvd issues? Is there no
>>> wbinvd in the bios shipped with older kvm? Which VM extension did you
>>> test, both Intel and AMD? I would bet that your X issues are due to the
>>> same effect. X startup/shutdown involves a lot of wbinvd calls on my
>>> test boxes.
>>>
>>>   
>>>       
>> kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
>> issues there. 
>>     
>
> So kvm can then simply ignore the instructions, or what does it do about
> them? I recall some replies in the related thread that some more work
> may be required.
>
>   

It currently ignores the instructions.  When device passthrough is 
implemented more work will be needed.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Dor Laor]

David Brown wrote:
> Uhm, not sure who to send this too...
>
> I thought I'd try out the realtime patch set and it didn't work at all
> with kvm. The console didn't dump anything and the system completely
> locked up.
>
> Anyone have any suggestions as to how to get more output on this issue?
>
> It got to the point of bringing up the tap interface and attaching it
> to the bridge but that was about it for the console messages.
>
> Thanks,
> - David Brown
>
>   
I tried to recreate your problem using 2.6.23-1 and latest rt patch (rt5).
The problem is that the kernel is not stable at all, I can't even 
compile the code over vnc -
my connection is constantly lost. So it might not be kvm problem?
Can you try is with -no-kvm and see if it's working - then its just a 
regular userspace process.

Anyway if all other things are stable on your end, can you send us 
dmesg/strace outputs?
Also try without the good -no-kvm-irqchip.
Regards,
Dor.
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>