2.6.23.1-rt4 and kvm 48

2.6.23.1-rt4 and kvm 48

[David Brown]

Uhm, not sure who to send this too...

I thought I'd try out the realtime patch set and it didn't work at all
with kvm. The console didn't dump anything and the system completely
locked up.

Anyone have any suggestions as to how to get more output on this issue?

It got to the point of bringing up the tap interface and attaching it
to the bridge but that was about it for the console messages.

Thanks,
- David Brown

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

David Brown wrote:
> Uhm, not sure who to send this too...
>
> I thought I'd try out the realtime patch set and it didn't work at all
> with kvm. The console didn't dump anything and the system completely
> locked up.
>
> Anyone have any suggestions as to how to get more output on this issue?
>   

Make sure CONFIG_PREEMPT_NOTIFIERS is enabled. Jan Kiszka reported 
success with a similar configuration.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Carsten Emde]

Avi Kivity wrote:
> David Brown wrote:
>> I thought I'd try out the realtime patch set and it didn't work at all
>> with kvm. The console didn't dump anything and the system completely
>> locked up.
Up to now, the unmodified kvm module never worked with any RT kernel. 
This would only change, if RT patched kernels were routinely used in the 
kvm release tests which is, for the time being, not the case.

However, tglx once provided a hand-crafted kvm version that can be used 
together with a selected RT kernel -> see "Latest Stable" 
(http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel.0.html) 
realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.

We used these modules to carry out a number of "proof of concept" tests 
which were very successful. The realtime capabilities of the host system 
remained unchanged irrespective of whether the kvm guest system (no 
graphics) was running or not. This was even the case while the guest 
system was booting.

We are planning to provide another "Latest Stable" realtime kernel 
within the next month or so. It will probably be based on kernel 2.6.23.

	--cbe

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Jan Kiszka]

Carsten Emde wrote:
> Avi Kivity wrote:
>> David Brown wrote:
>>> I thought I'd try out the realtime patch set and it didn't work at all
>>> with kvm. The console didn't dump anything and the system completely
>>> locked up.
> Up to now, the unmodified kvm module never worked with any RT kernel. 
> This would only change, if RT patched kernels were routinely used in the 
> kvm release tests which is, for the time being, not the case.

As far as I understood (but didn't re-test yet), latest kvm release does
include all features required to run over -rt. This was at least the
case for kvm-git I picked up last week and merged into 2.6.23-rt1. The
only issue is that out-of-tree kvm doesn't enable the preemption
notifiers (maybe there were one or two further minor issues, I think to
recall missing smp_ops exports being one, but these were not -rt related).

> 
> However, tglx once provided a hand-crafted kvm version that can be used 
> together with a selected RT kernel -> see "Latest Stable" 
> (http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel.0.html) 
> realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.
> 
> We used these modules to carry out a number of "proof of concept" tests 
> which were very successful. The realtime capabilities of the host system 
> remained unchanged irrespective of whether the kvm guest system (no 
> graphics) was running or not. This was even the case while the guest 
> system was booting.

Interesting result - you've read about the wbinvd issues? Is there no
wbinvd in the bios shipped with older kvm? Which VM extension did you
test, both Intel and AMD? I would bet that your X issues are due to the
same effect. X startup/shutdown involves a lot of wbinvd calls on my
test boxes.

To sum up my findings: kvm over whatever RT kernel is risky
(latency-wise) unless you can audit or even para-virtualise your guest
OS /wrt wbinvd. Maybe we can discuss this also in Linz by the end of the
week. :)

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Alexander Graf]

On Oct 30, 2007, at 10:49 AM, Jan Kiszka wrote:

> Carsten Emde wrote:
>> Avi Kivity wrote:
>>> David Brown wrote:
>>>> I thought I'd try out the realtime patch set and it didn't work  
>>>> at all
>>>> with kvm. The console didn't dump anything and the system  
>>>> completely
>>>> locked up.
>> Up to now, the unmodified kvm module never worked with any RT kernel.
>> This would only change, if RT patched kernels were routinely used  
>> in the
>> kvm release tests which is, for the time being, not the case.
>
> As far as I understood (but didn't re-test yet), latest kvm release  
> does
> include all features required to run over -rt. This was at least the
> case for kvm-git I picked up last week and merged into 2.6.23-rt1. The
> only issue is that out-of-tree kvm doesn't enable the preemption
> notifiers (maybe there were one or two further minor issues, I  
> think to
> recall missing smp_ops exports being one, but these were not -rt  
> related).
>
>>
>> However, tglx once provided a hand-crafted kvm version that can be  
>> used
>> together with a selected RT kernel -> see "Latest Stable"
>> (http://www.osadl.org/Latest-Stable.latest-stable-realtime-kernel. 
>> 0.html)
>> realtime kernel. It is based on kernel 2.6.21.6-rt21 and kvm-28.
>>
>> We used these modules to carry out a number of "proof of concept"  
>> tests
>> which were very successful. The realtime capabilities of the host  
>> system
>> remained unchanged irrespective of whether the kvm guest system (no
>> graphics) was running or not. This was even the case while the guest
>> system was booting.
>
> Interesting result - you've read about the wbinvd issues? Is there no
> wbinvd in the bios shipped with older kvm? Which VM extension did you
> test, both Intel and AMD? I would bet that your X issues are due to  
> the
> same effect. X startup/shutdown involves a lot of wbinvd calls on my
> test boxes.
>
> To sum up my findings: kvm over whatever RT kernel is risky
> (latency-wise) unless you can audit or even para-virtualise your guest
> OS /wrt wbinvd. Maybe we can discuss this also in Linz by the end  
> of the
> week. :)

As I will be in Linz by the end of the week as well, could you please  
try to do any coordination as to when you meet on the lists (or CC  
me)? I'd like to get the chance to join you on this :-).

Thanks,

Alex

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

Jan Kiszka wrote:
> Interesting result - you've read about the wbinvd issues? Is there no
> wbinvd in the bios shipped with older kvm? Which VM extension did you
> test, both Intel and AMD? I would bet that your X issues are due to the
> same effect. X startup/shutdown involves a lot of wbinvd calls on my
> test boxes.
>
>   

kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
issues there.  I hope that Intel will add similar traps in future 
iterations of VT.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Jan Kiszka]

Avi Kivity wrote:
> Jan Kiszka wrote:
>> Interesting result - you've read about the wbinvd issues? Is there no
>> wbinvd in the bios shipped with older kvm? Which VM extension did you
>> test, both Intel and AMD? I would bet that your X issues are due to the
>> same effect. X startup/shutdown involves a lot of wbinvd calls on my
>> test boxes.
>>
>>   
> 
> kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
> issues there. 

So kvm can then simply ignore the instructions, or what does it do about
them? I recall some replies in the related thread that some more work
may be required.

> I hope that Intel will add similar traps in future 
> iterations of VT.

Yep.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: [kvm-devel] 2.6.23.1-rt4 and kvm 48

[Avi Kivity]

Jan Kiszka wrote:
> Avi Kivity wrote:
>   
>> Jan Kiszka wrote:
>>     
>>> Interesting result - you've read about the wbinvd issues? Is there no
>>> wbinvd in the bios shipped with older kvm? Which VM extension did you
>>> test, both Intel and AMD? I would bet that your X issues are due to the
>>> same effect. X startup/shutdown involves a lot of wbinvd calls on my
>>> test boxes.
>>>
>>>   
>>>       
>> kvm.git now traps invd and wbinvd on amd, so there shouldn't be any 
>> issues there. 
>>     
>
> So kvm can then simply ignore the instructions, or what does it do about
> them? I recall some replies in the related thread that some more work
> may be required.
>
>   

It currently ignores the instructions.  When device passthrough is 
implemented more work will be needed.


-- 
Any sufficiently difficult bug is indistinguishable from a feature.

Re: 2.6.23.1-rt4 and kvm 48

[Dor Laor]

David Brown wrote:
> Uhm, not sure who to send this too...
>
> I thought I'd try out the realtime patch set and it didn't work at all
> with kvm. The console didn't dump anything and the system completely
> locked up.
>
> Anyone have any suggestions as to how to get more output on this issue?
>
> It got to the point of bringing up the tap interface and attaching it
> to the bridge but that was about it for the console messages.
>
> Thanks,
> - David Brown
>
>   
I tried to recreate your problem using 2.6.23-1 and latest rt patch (rt5).
The problem is that the kernel is not stable at all, I can't even 
compile the code over vnc -
my connection is constantly lost. So it might not be kvm problem?
Can you try is with -no-kvm and see if it's working - then its just a 
regular userspace process.

Anyway if all other things are stable on your end, can you send us 
dmesg/strace outputs?
Also try without the good -no-kvm-irqchip.
Regards,
Dor.
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>
>   


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[Carsten Emde]

Dor,

>> [..]
>> It got to the point of bringing up the tap interface and attaching it
>> to the bridge but that was about it for the console messages.
> I tried to recreate your problem using 2.6.23-1 and latest rt patch (rt5).
> The problem is that the kernel is not stable at all, I can't even 
> compile the code over vnc -
> my connection is constantly lost. So it might not be kvm problem?
> Can you try is with -no-kvm and see if it's working - then its just a 
> regular userspace process.
> Anyway if all other things are stable on your end, can you send us 
> dmesg/strace outputs?
Thanks, Dor, for offering help.

The most recent RT-Preempt patch (2.6.23.1-rt5) still has an SMP issue 
which need to be sorted out first.

When a working RT-Preempt patch is available, I will be glad to send 
dmesg outputs. We may then share our activities to create the next 
"Latest stable" RT-Preempt kernel that includes kvm support.

	--cbe

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[David Brown]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

> > I tried to recreate your problem using 2.6.23-1 and latest rt patch (rt5).
> > The problem is that the kernel is not stable at all, I can't even
> > compile the code over vnc -
> > my connection is constantly lost. So it might not be kvm problem?
> > Can you try is with -no-kvm and see if it's working - then its just a
> > regular userspace process.
> > Anyway if all other things are stable on your end, can you send us
> > dmesg/strace outputs?
> Thanks, Dor, for offering help.
>
> The most recent RT-Preempt patch (2.6.23.1-rt5) still has an SMP issue
> which need to be sorted out first.
>
> When a working RT-Preempt patch is available, I will be glad to send
> dmesg outputs. We may then share our activities to create the next
> "Latest stable" RT-Preempt kernel that includes kvm support.

Wow, that smp issue may be a problem I've got a Intel Core 2 cpu with
just 2 cores.

but I've not seen a dump like this in a while (lustre?).

I've updated to 2.6.23-rt5 and kvm-50 and I got this dump....

[  254.639143] audit(1194322621.379:3): dev=tap0 prom=256 old_prom=0 auid=429496
7295
[  254.647231] br0: port 2(tap0) entering learning state
[  255.988741] IRQ-23[942]: segfault at 0000000000000000 rip 00002ac.994386] Cal
l Trace:
<4>[  255.994387] <3>[  255.998090] BUG: scheduling while atomic: swapper/0x0  2
56.002986] Call Trace:
[  256.002986]
[  256.00ad!
<4>[  256.007635]
<4>[  256.007635] Call Trace:G: scheduling while atomic: swapper/0x00000003/0, C
PU#all Trace:
[  256.016955]
[  256.016956] bad: sched1615]
<4>[  256.021615] Call Trace:
[  256.021616]e atomic: swapper/0x00000003/0, CPU#1
[  256.026276] .030935]
[  256.030936] bad: scheduling from the idl35595] Call Trace:
[  256.035595]
[  256.035596] B0000003/0, CPU#1
<4>[  256.040256]

This loops forever basically...
I'm really confused about this one. I've got the full minicom.cap file
from the session that produced this its attached zipped up.

If you'd like any other information I'd be glad to post it.

- David Brown

[-- Attachment #2: minicom.cap.gz --]
[-- Type: application/x-gzip, Size: 32078 bytes --]

[-- Attachment #3: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: 2.6.23.1-rt4 and kvm 48

[Dor Laor]

[-- Attachment #1.1: Type: text/plain, Size: 2307 bytes --]

David Brown wrote:
>>> I tried to recreate your problem using 2.6.23-1 and latest rt patch (rt5).
>>> The problem is that the kernel is not stable at all, I can't even
>>> compile the code over vnc -
>>> my connection is constantly lost. So it might not be kvm problem?
>>> Can you try is with -no-kvm and see if it's working - then its just a
>>> regular userspace process.
>>> Anyway if all other things are stable on your end, can you send us
>>> dmesg/strace outputs?
>>>       
>> Thanks, Dor, for offering help.
>>
>> The most recent RT-Preempt patch (2.6.23.1-rt5) still has an SMP issue
>> which need to be sorted out first.
>>
>> When a working RT-Preempt patch is available, I will be glad to send
>> dmesg outputs. We may then share our activities to create the next
>> "Latest stable" RT-Preempt kernel that includes kvm support.
>>     
>
> Wow, that smp issue may be a problem I've got a Intel Core 2 cpu with
> just 2 cores.
>
> but I've not seen a dump like this in a while (lustre?).
>
> I've updated to 2.6.23-rt5 and kvm-50 and I got this dump....
>
> [  254.639143] audit(1194322621.379:3): dev=tap0 prom=256 old_prom=0 auid=429496
> 7295
> [  254.647231] br0: port 2(tap0) entering learning state
> [  255.988741] IRQ-23[942]: segfault at 0000000000000000 rip 00002ac.994386] Cal
> l Trace:
> <4>[  255.994387] <3>[  255.998090] BUG: scheduling while atomic: swapper/0x0  2
> 56.002986] Call Trace:
> [  256.002986]
> [  256.00ad!
> <4>[  256.007635]
> <4>[  256.007635] Call Trace:G: scheduling while atomic: swapper/0x00000003/0, C
> PU#all Trace:
> [  256.016955]
> [  256.016956] bad: sched1615]
> <4>[  256.021615] Call Trace:
> [  256.021616]e atomic: swapper/0x00000003/0, CPU#1
> [  256.026276] .030935]
> [  256.030936] bad: scheduling from the idl35595] Call Trace:
> [  256.035595]
> [  256.035596] B0000003/0, CPU#1
> <4>[  256.040256]
>
> This loops forever basically...
> I'm really confused about this one. I've got the full minicom.cap file
> from the session that produced this its attached zipped up.
>
> If you'd like any other information I'd be glad to post it.
>
> - David Brown
>   
If the rt patch is not smp stable you can boot with single cpu and run 
kvm there.
Maybe just pinning the guest to specific cpu will help although I had 
troubles with X too.
Dor.

[-- Attachment #1.2: Type: text/html, Size: 2742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: 2.6.23.1-rt4 and kvm 48

[Carsten Emde]

Dor,

>>>> I tried to recreate your problem using 2.6.23-1 and latest rt patch 
>>>> (rt5).
>>>> The problem is that the kernel is not stable at all, I can't even
>>>> compile the code over vnc -
>>>> my connection is constantly lost. So it might not be kvm problem?
>>>> Can you try is with -no-kvm and see if it's working - then its just a
>>>> regular userspace process.
>>>> Anyway if all other things are stable on your end, can you send us
>>>> dmesg/strace outputs?
>>> Thanks, Dor, for offering help.
>>> The most recent RT-Preempt patch (2.6.23.1-rt5) still has an SMP issue
>>> which need to be sorted out first.
>>> When a working RT-Preempt patch is available, I will be glad to send
>>> dmesg outputs. We may then share our activities to create the next
>>> "Latest stable" RT-Preempt kernel that includes kvm support.
>> Wow, that smp issue may be a problem I've got a Intel Core 2 cpu with
>> just 2 cores.
>> but I've not seen a dump like this in a while (lustre?).
We see this all the time when we try to run unmodified kvm in an 
rt-kernel, nothing special.

>> I've updated to 2.6.23-rt5 and kvm-50 and I got this dump....
>> [..]
>> BUG: scheduling while atomic: 
>> [..]
>> This loops forever basically...
>> I'm really confused about this one. I've got the full minicom.cap file
>> from the session that produced this its attached zipped up.
>> If you'd like any other information I'd be glad to post it.
No, please not. Our log files are full of it. The "BUG: scheduling while 
atomic" problem is not related to the current SMP issue. It is the 
result of running rt-unaware code in an rt-kernel.

> If the rt patch is not smp stable you can boot with single cpu and run 
> kvm there.
> Maybe just pinning the guest to specific cpu will help although I had 
> troubles with X too.
This does not fix the problem. Save your energy. We live with this 
situation since kvm-28. Please wait until 2.6.23.<x>-rt<y> is stable. 
Thereafter, let's create together a kvm version that works in an 
rt-kernel and let's make sure that from this moment onwards subsequent 
kvm upgrade changes are tested in an rt-kernel before being released.

	--cbe

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: 2.6.23.1-rt4 and kvm 48

[David Brown]

On 11/6/07, Carsten Emde <Carsten.Emde-Q945KHDl0DbYtjvyW6yDsg@public.gmane.org> wrote:
> Dor,
>
> >>>> I tried to recreate your problem using 2.6.23-1 and latest rt patch
> >>>> (rt5).
> >>>> The problem is that the kernel is not stable at all, I can't even
> >>>> compile the code over vnc -
> >>>> my connection is constantly lost. So it might not be kvm problem?
> >>>> Can you try is with -no-kvm and see if it's working - then its just a
> >>>> regular userspace process.
> >>>> Anyway if all other things are stable on your end, can you send us
> >>>> dmesg/strace outputs?
> >>> Thanks, Dor, for offering help.
> >>> The most recent RT-Preempt patch (2.6.23.1-rt5) still has an SMP issue
> >>> which need to be sorted out first.
> >>> When a working RT-Preempt patch is available, I will be glad to send
> >>> dmesg outputs. We may then share our activities to create the next
> >>> "Latest stable" RT-Preempt kernel that includes kvm support.
> >> Wow, that smp issue may be a problem I've got a Intel Core 2 cpu with
> >> just 2 cores.
> >> but I've not seen a dump like this in a while (lustre?).
> We see this all the time when we try to run unmodified kvm in an
> rt-kernel, nothing special.
>
> >> I've updated to 2.6.23-rt5 and kvm-50 and I got this dump....
> >> [..]
> >> BUG: scheduling while atomic:
> >> [..]
> >> This loops forever basically...
> >> I'm really confused about this one. I've got the full minicom.cap file
> >> from the session that produced this its attached zipped up.
> >> If you'd like any other information I'd be glad to post it.
> No, please not. Our log files are full of it. The "BUG: scheduling while
> atomic" problem is not related to the current SMP issue. It is the
> result of running rt-unaware code in an rt-kernel.
>
> > If the rt patch is not smp stable you can boot with single cpu and run
> > kvm there.
> > Maybe just pinning the guest to specific cpu will help although I had
> > troubles with X too.
> This does not fix the problem. Save your energy. We live with this
> situation since kvm-28. Please wait until 2.6.23.<x>-rt<y> is stable.
> Thereafter, let's create together a kvm version that works in an
> rt-kernel and let's make sure that from this moment onwards subsequent
> kvm upgrade changes are tested in an rt-kernel before being released.

I can certainly help with the testing part applying patches and such
is easy. Not too knowledgeable about how the realtime kernel works. If
I were paid to work on that I'd probably make a better effort to know
those things, but currently I only have the time to test...

- David Brown

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/