VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Koehrer Mathias (ETAS/ESS2)]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

Hi all,

I have detected the following issue:
I use an x86 PC using kernel 3.2.34-rt51 and I want to run VirtualBox 4.2.4 using this PC as host.
However, the PC freezes completely when loading the vbox driver "vboxdrv".
Using the very same kernel without the RT_PREEMPT patch shows no issues.

Using an older version of VirtualBox (V4.1.22) works fine as well.

I know that this might be slightly off-topic. However there might be some RT_PREEMPT users that have detected the same issue and know a solution for this.

I have attached my kernel .config file.

Thanks for any feedback on this.

Regards

Mathias



[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 13945 bytes --]

Re: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On 2012-12-14 09:58, Koehrer Mathias (ETAS/ESS2) wrote:
> Hi all,
> 
> I have detected the following issue:
> I use an x86 PC using kernel 3.2.34-rt51 and I want to run VirtualBox 4.2.4 using this PC as host.

> However, the PC freezes completely when loading the vbox driver "vboxdrv".
> Using the very same kernel without the RT_PREEMPT patch shows no issues.
> 
> Using an older version of VirtualBox (V4.1.22) works fine as well.
> 
> I know that this might be slightly off-topic. However there might be some RT_PREEMPT users that have detected the same issue and know a solution for this.
> 
> I have attached my kernel .config file.
> 
> Thanks for any feedback on this.

Try enabling lock debugging. Likely there is an incompatibility in the
vbox driver or even a bug, and that may reveal more details.

Otherwise: KVM is the better choice on Linux hosts, and it works fine
over -rt.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Koehrer Mathias (ETAS/ESS2)]

[-- Attachment #1: Type: text/plain, Size: 2538 bytes --]

Hi all,

> >
> > I have detected the following issue:
> > I use an x86 PC using kernel 3.2.34-rt51 and I want to run VirtualBox 4.2.4 using
> this PC as host.
> 
> > However, the PC freezes completely when loading the vbox driver "vboxdrv".
> > Using the very same kernel without the RT_PREEMPT patch shows no issues.
> >
> > Using an older version of VirtualBox (V4.1.22) works fine as well.
> >
> > I know that this might be slightly off-topic. However there might be some
> RT_PREEMPT users that have detected the same issue and know a solution for this.
> >
> > I have attached my kernel .config file.
> >
> > Thanks for any feedback on this.
> 
> Try enabling lock debugging. Likely there is an incompatibility in the
> vbox driver or even a bug, and that may reveal more details.
> 
> Otherwise: KVM is the better choice on Linux hosts, and it works fine
> over -rt.
I am about to try this out in parallel. It looks promising...


In addition, I did a long debug session (using a parallel port LED adapter any many PC resets...) to identify the issue.
Lock debugging did not work as really everything was frozen or the PC.
However, I finally identified the critical lines of code.
The relevant source file is the attached file "spinlock-r0drv-linux.c" of the VirtualBox OSE V4.2.4.
The freeze happens in line 158 (spin_unlock_irqrestore()).

----------- BEGIN CODE FRAGMENT --------------------
RTDECL(void) RTSpinlockRelease(RTSPINLOCK Spinlock)
{
    PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
    RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE_VARS();
    AssertMsg(pThis && pThis->u32Magic == RTSPINLOCK_MAGIC,
              ("pThis=%p u32Magic=%08x\n", pThis, pThis ? (int)pThis->u32Magic : 0));
    RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE(pThis);

    if (pThis->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)
    {
        unsigned long fIntSaved = pThis->fIntSaved;
        pThis->fIntSaved = 0;
        spin_unlock_irqrestore(&pThis->Spinlock, fIntSaved);   // <--- Here is the freeze!
    }
    else
        spin_unlock(&pThis->Spinlock);

    RT_ASSERT_PREEMPT_CPUID();
}
RT_EXPORT_SYMBOL(RTSpinlockRelease);
----------- END CODE FRAGMENT --------------------

It has something to do with the spin locks used in VirtualBox.
Likely it does not consider the RT_PREEMPT use case here...
Is there a way to make the spin lock calls compatible to the RT_PREEMPT patch?

Any ideas for solving this issue are highly welcome!

Thanks a lot.
Best regards.

Mathias


[-- Attachment #2: spinlock-r0drv-linux.c --]
[-- Type: text/plain, Size: 5729 bytes --]

/* $Id: spinlock-r0drv-linux.c $ */
/** @file
 * IPRT - Spinlocks, Ring-0 Driver, Linux.
 */

/*
 * Copyright (C) 2006-2010 Oracle Corporation
 *
 * This file is part of VirtualBox Open Source Edition (OSE), as
 * available from http://www.virtualbox.org. This file is free software;
 * you can redistribute it and/or modify it under the terms of the GNU
 * General Public License (GPL) as published by the Free Software
 * Foundation, in version 2 as it comes in the "COPYING" file of the
 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
 *
 * The contents of this file may alternatively be used under the terms
 * of the Common Development and Distribution License Version 1.0
 * (CDDL) only, as it comes in the "COPYING.CDDL" file of the
 * VirtualBox OSE distribution, in which case the provisions of the
 * CDDL are applicable instead of those of the GPL.
 *
 * You may elect to license modified versions of this file under the
 * terms and conditions of either the GPL or the CDDL or both.
 */


/*******************************************************************************
*   Header Files                                                               *
*******************************************************************************/
#include "the-linux-kernel.h"
#include "internal/iprt.h"
#include <iprt/spinlock.h>

#include <iprt/asm.h>
#if defined(RT_ARCH_AMD64) || defined(RT_ARCH_X86)
# include <iprt/asm-amd64-x86.h>
#endif
#include <iprt/assert.h>
#include <iprt/err.h>
#include <iprt/mem.h>
#include <iprt/mp.h>
#include <iprt/thread.h>
#include "internal/magics.h"


/*******************************************************************************
*   Structures and Typedefs                                                    *
*******************************************************************************/
/**
 * Wrapper for the spinlock_t structure.
 */
typedef struct RTSPINLOCKINTERNAL
{
    /** Spinlock magic value (RTSPINLOCK_MAGIC). */
    uint32_t volatile       u32Magic;
    /** The spinlock creation flags.  */
    uint32_t                fFlags;
    /** The saved interrupt flag. */
    unsigned long volatile  fIntSaved;
    /** The linux spinlock structure. */
    spinlock_t              Spinlock;
#ifdef RT_MORE_STRICT
    /** The idAssertCpu variable before acquring the lock for asserting after
     *  releasing the spinlock. */
    RTCPUID volatile        idAssertCpu;
    /** The CPU that owns the lock. */
    RTCPUID volatile        idCpuOwner;
#endif
} RTSPINLOCKINTERNAL, *PRTSPINLOCKINTERNAL;



RTDECL(int)  RTSpinlockCreate(PRTSPINLOCK pSpinlock, uint32_t fFlags, const char *pszName)
{
    PRTSPINLOCKINTERNAL pThis;
    AssertReturn(fFlags == RTSPINLOCK_FLAGS_INTERRUPT_SAFE || fFlags == RTSPINLOCK_FLAGS_INTERRUPT_UNSAFE, VERR_INVALID_PARAMETER);

    /*
     * Allocate.
     */
    Assert(sizeof(RTSPINLOCKINTERNAL) > sizeof(void *));
    pThis = (PRTSPINLOCKINTERNAL)RTMemAlloc(sizeof(*pThis));
    if (!pThis)
        return VERR_NO_MEMORY;
    /*
     * Initialize and return.
     */
    pThis->u32Magic     = RTSPINLOCK_MAGIC;
    pThis->fFlags       = fFlags;
    pThis->fIntSaved    = 0;
#ifdef RT_MORE_STRICT
    pThis->idCpuOwner   = NIL_RTCPUID;
    pThis->idAssertCpu  = NIL_RTCPUID;
#endif
    spin_lock_init(&pThis->Spinlock);

    *pSpinlock = pThis;
    return VINF_SUCCESS;
}
RT_EXPORT_SYMBOL(RTSpinlockCreate);


RTDECL(int)  RTSpinlockDestroy(RTSPINLOCK Spinlock)
{
    /*
     * Validate input.
     */
    PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
    if (!pThis)
        return VERR_INVALID_PARAMETER;
    if (pThis->u32Magic != RTSPINLOCK_MAGIC)
    {
        AssertMsgFailed(("Invalid spinlock %p magic=%#x\n", pThis, pThis->u32Magic));
        return VERR_INVALID_PARAMETER;
    }

    ASMAtomicIncU32(&pThis->u32Magic);
    RTMemFree(pThis);
    return VINF_SUCCESS;
}
RT_EXPORT_SYMBOL(RTSpinlockDestroy);


RTDECL(void) RTSpinlockAcquire(RTSPINLOCK Spinlock)
{
    PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
    RT_ASSERT_PREEMPT_CPUID_VAR();
    AssertMsg(pThis && pThis->u32Magic == RTSPINLOCK_MAGIC,
              ("pThis=%p u32Magic=%08x\n", pThis, pThis ? (int)pThis->u32Magic : 0));

    if (pThis->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)
    {
        unsigned long fIntSaved;
        spin_lock_irqsave(&pThis->Spinlock, fIntSaved);
        pThis->fIntSaved = fIntSaved;
    }
    else
        spin_lock(&pThis->Spinlock);

    RT_ASSERT_PREEMPT_CPUID_SPIN_ACQUIRED(pThis);
}
RT_EXPORT_SYMBOL(RTSpinlockAcquire);


RTDECL(void) RTSpinlockRelease(RTSPINLOCK Spinlock)
{
    PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
    RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE_VARS();
    AssertMsg(pThis && pThis->u32Magic == RTSPINLOCK_MAGIC,
              ("pThis=%p u32Magic=%08x\n", pThis, pThis ? (int)pThis->u32Magic : 0));
    RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE(pThis);

    if (pThis->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)
    {
        unsigned long fIntSaved = pThis->fIntSaved;
        pThis->fIntSaved = 0;
        spin_unlock_irqrestore(&pThis->Spinlock, fIntSaved);
    }
    else
        spin_unlock(&pThis->Spinlock);

    RT_ASSERT_PREEMPT_CPUID();
}
RT_EXPORT_SYMBOL(RTSpinlockRelease);


RTDECL(void) RTSpinlockReleaseNoInts(RTSPINLOCK Spinlock)
{
#if 1
    if (RT_UNLIKELY(!(Spinlock->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)))
        RTAssertMsg2("RTSpinlockReleaseNoInts: %p (magic=%#x)\n", Spinlock, Spinlock->u32Magic);
#else
    AssertRelease(Spinlock->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE);
#endif
    RTSpinlockRelease(Spinlock);
}
RT_EXPORT_SYMBOL(RTSpinlockReleaseNoInts);

Re: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Thomas Gleixner]

On Wed, 19 Dec 2012, Koehrer Mathias (ETAS/ESS2) wrote:
> > >
> > > I have detected the following issue:
> > > I use an x86 PC using kernel 3.2.34-rt51 and I want to run VirtualBox 4.2.4 using
> > this PC as host.
> > 
> > > However, the PC freezes completely when loading the vbox driver "vboxdrv".
> > > Using the very same kernel without the RT_PREEMPT patch shows no issues.
> > >
> > > Using an older version of VirtualBox (V4.1.22) works fine as well.
> > >
> > > I know that this might be slightly off-topic. However there might be some
> > RT_PREEMPT users that have detected the same issue and know a solution for this.
> > >
> > > I have attached my kernel .config file.
> > >
> > > Thanks for any feedback on this.
> > 
> > Try enabling lock debugging. Likely there is an incompatibility in the
> > vbox driver or even a bug, and that may reveal more details.
> > 
> > Otherwise: KVM is the better choice on Linux hosts, and it works fine
> > over -rt.
> I am about to try this out in parallel. It looks promising...
> 
> 
> In addition, I did a long debug session (using a parallel port LED adapter any many PC resets...) to identify the issue.
> Lock debugging did not work as really everything was frozen or the PC.
> However, I finally identified the critical lines of code.
> The relevant source file is the attached file "spinlock-r0drv-linux.c" of the VirtualBox OSE V4.2.4.
> The freeze happens in line 158 (spin_unlock_irqrestore()).
> 
> ----------- BEGIN CODE FRAGMENT --------------------
> RTDECL(void) RTSpinlockRelease(RTSPINLOCK Spinlock)
> {
>     PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
>     RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE_VARS();
>     AssertMsg(pThis && pThis->u32Magic == RTSPINLOCK_MAGIC,
>               ("pThis=%p u32Magic=%08x\n", pThis, pThis ? (int)pThis->u32Magic : 0));
>     RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE(pThis);
> 
>     if (pThis->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)
>     {
>         unsigned long fIntSaved = pThis->fIntSaved;
>         pThis->fIntSaved = 0;
>         spin_unlock_irqrestore(&pThis->Spinlock, fIntSaved);   // <--- Here is the freeze!
>     }
>     else
>         spin_unlock(&pThis->Spinlock);
> 
>     RT_ASSERT_PREEMPT_CPUID();
> }
> RT_EXPORT_SYMBOL(RTSpinlockRelease);
> ----------- END CODE FRAGMENT --------------------

Was it really necessary to expose us to the camel case inflicted
horror of the VirtualBox spinlock abstraction layer just a few days
before Christmas?

> It has something to do with the spin locks used in VirtualBox.

Why am I not surprised?

> Likely it does not consider the RT_PREEMPT use case here...

I don't think so. That's a genuine problem unrelated to RT.

> Is there a way to make the spin lock calls compatible to the RT_PREEMPT patch?

No. The API usage seems to be correct, but RT might expose an
otherwise hard to trigger live time issue, data corruption or whatever
problem.

You said:
> Lock debugging did not work as really everything was frozen or the PC.

That's an indicator that something is really wrong. Can you just run
the same crap on a non-rt kernel with CONFIG_PROVE_LOCKING enabled ?

Thanks,

	tglx

AW: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Koehrer Mathias (ETAS/ESS2)]

[-- Attachment #1: Type: text/plain, Size: 5989 bytes --]

Hi Thomas,

thanks for the response.
> On Wed, 19 Dec 2012, Koehrer Mathias (ETAS/ESS2) wrote:
> > > >
> > > > I have detected the following issue:
> > > > I use an x86 PC using kernel 3.2.34-rt51 and I want to run VirtualBox 4.2.4
> using
> > > this PC as host.
> > >
> > > > However, the PC freezes completely when loading the vbox driver "vboxdrv".
> > > > Using the very same kernel without the RT_PREEMPT patch shows no issues.
> > > >
> > > > Using an older version of VirtualBox (V4.1.22) works fine as well.
> > > >
> > > > I know that this might be slightly off-topic. However there might be some
> > > RT_PREEMPT users that have detected the same issue and know a solution for
> this.
> > >
> > > (...)
> > > Try enabling lock debugging. Likely there is an incompatibility in the
> > > vbox driver or even a bug, and that may reveal more details.
> >
> > In addition, I did a long debug session (using a parallel port LED adapter any
> many PC resets...) to identify the issue.
> > Lock debugging did not work as really everything was frozen or the PC.
> > However, I finally identified the critical lines of code.
> > The relevant source file is the attached file "spinlock-r0drv-linux.c" of the
> VirtualBox OSE V4.2.4.
> > The freeze happens in line 158 (spin_unlock_irqrestore()).
> >
> > ----------- BEGIN CODE FRAGMENT --------------------
> > RTDECL(void) RTSpinlockRelease(RTSPINLOCK Spinlock)
> > {
> >     PRTSPINLOCKINTERNAL pThis = (PRTSPINLOCKINTERNAL)Spinlock;
> >     RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE_VARS();
> >     AssertMsg(pThis && pThis->u32Magic == RTSPINLOCK_MAGIC,
> >               ("pThis=%p u32Magic=%08x\n", pThis, pThis ? (int)pThis->u32Magic :
> 0));
> >     RT_ASSERT_PREEMPT_CPUID_SPIN_RELEASE(pThis);
> >
> >     if (pThis->fFlags & RTSPINLOCK_FLAGS_INTERRUPT_SAFE)
> >     {
> >         unsigned long fIntSaved = pThis->fIntSaved;
> >         pThis->fIntSaved = 0;
> >         spin_unlock_irqrestore(&pThis->Spinlock, fIntSaved);   // <--- Here is the
> freeze!
> >     }
> >     else
> >         spin_unlock(&pThis->Spinlock);
> >
> >     RT_ASSERT_PREEMPT_CPUID();
> > }
> > RT_EXPORT_SYMBOL(RTSpinlockRelease);
> > ----------- END CODE FRAGMENT --------------------
> 
> Was it really necessary to expose us to the camel case inflicted
> horror of the VirtualBox spinlock abstraction layer just a few days
> before Christmas?
Sorry for  that!

> You said:
> > Lock debugging did not work as really everything was frozen or the PC.
> 
> That's an indicator that something is really wrong. Can you just run
> the same crap on a non-rt kernel with CONFIG_PROVE_LOCKING enabled ?

I did. It shows actually an error.
Here is the relevant dmesg output:

--------------- BEGIN dmesg output -----------------
vboxdrv: Found 4 processor cores.
vboxdrv: fAsync=0 offMin=0x50c offMax=0x5940

=================================
[ INFO: inconsistent lock state ]
3.2.34-2 #1
---------------------------------
inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
swapper/1/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
 (&(&pThis->Spinlock)->rlock){?.+...}, at: [<f883c7a9>] VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
{HARDIRQ-ON-W} state was registered at:
  [<c104b99a>] __lock_acquire+0x509/0x1337
  [<c104cb3a>] lock_acquire+0x42/0x59
  [<c13204d5>] _raw_spin_lock+0x25/0x34
  [<f883c7b3>] VBoxHost_RTSpinlockAcquire+0x1f/0x24 [vboxdrv]
  [<f883986f>] VBoxHost_RTMpNotificationRegister+0x36/0x11a [vboxdrv]
  [<f88324f8>] supdrvInitDevExt+0x4f4/0x65c [vboxdrv]
  [<f8868053>] VBoxDrvLinuxInit+0x53/0xcd [vboxdrv]
  [<c100106b>] do_one_initcall+0x6b/0x10c
  [<c1054299>] sys_init_module+0x122f/0x13fa
  [<c1321990>] sysenter_do_call+0x12/0x36
irq event stamp: 278096
hardirqs last  enabled at (278093): [<c1007c3a>] mwait_idle+0x57/0x7c
hardirqs last disabled at (278094): [<c1320f34>] call_function_interrupt+0x28/0x34
softirqs last  enabled at (278096): [<c102e3db>] _local_bh_enable+0xd/0xf
softirqs last disabled at (278095): [<c102e7a8>] irq_enter+0x29/0x4e

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&(&pThis->Spinlock)->rlock);
  <Interrupt>
    lock(&(&pThis->Spinlock)->rlock);

 *** DEADLOCK ***

no locks held by swapper/1/0.

stack backtrace:
Pid: 0, comm: swapper/1 Tainted: G           O 3.2.34-2 #1
Call Trace:
 [<c131c0bb>] print_usage_bug.part.27+0x1f0/0x1fa
 [<c104b313>] mark_lock+0x334/0x4b2
 [<c104a95b>] ? check_usage_backwards+0xce/0xce
 [<c104b92d>] __lock_acquire+0x49c/0x1337
 [<c104cb3a>] lock_acquire+0x42/0x59
 [<f883c7a9>] ? VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
 [<c1320583>] _raw_spin_lock_irqsave+0x2e/0x3e
 [<f883c7a9>] ? VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
 [<f883c7a9>] VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
 [<f8831d69>] ? supdrvGipMpEventOnline+0x14d/0x14d [vboxdrv]
 [<f8831c77>] supdrvGipMpEventOnline+0x5b/0x14d [vboxdrv]
 [<f8831d69>] ? supdrvGipMpEventOnline+0x14d/0x14d [vboxdrv]
 [<f8831d77>] supdrvGipInitOnCpu+0xe/0x10 [vboxdrv]
 [<f883b688>] rtmpLinuxWrapper+0x22/0x2d [vboxdrv]
 [<f883b666>] ? VBoxHost_RTMpCpuId+0xa/0xa [vboxdrv]
 [<c1051253>] generic_smp_call_function_interrupt+0x6b/0x112
 [<c1014595>] smp_call_function_interrupt+0x20/0x2e
 [<c1320f3b>] call_function_interrupt+0x2f/0x34
 [<c104007b>] ? __run_hrtimer.isra.28+0x33/0x9e
 [<c1007c42>] ? mwait_idle+0x5f/0x7c
 [<c10014c2>] cpu_idle+0x4d/0x76
 [<c14ff2e8>] start_secondary+0x1ab/0x1b2
vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
vboxdrv: Successfully loaded version 4.2.4 (interface 0x001a0004).
vboxpci: pci-stub module not available, cannot detach PCI devices
vboxpci: IOMMU not found (not compiled)
--------------- END dmesg output -----------------

I attached the kernel configuration used for this test.

Thanks for the support.

Regards and merry Christmas.

Mathias


[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 14433 bytes --]

Re: AW: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Steven Rostedt]

On Fri, 2012-12-21 at 08:23 +0000, Koehrer Mathias (ETAS/ESS2) wrote:

> > That's an indicator that something is really wrong. Can you just run
> > the same crap on a non-rt kernel with CONFIG_PROVE_LOCKING enabled ?
> 
> I did. It shows actually an error.
> Here is the relevant dmesg output:
> 
> --------------- BEGIN dmesg output -----------------
> vboxdrv: Found 4 processor cores.
> vboxdrv: fAsync=0 offMin=0x50c offMax=0x5940
> 
> =================================
> [ INFO: inconsistent lock state ]
> 3.2.34-2 #1
> ---------------------------------
> inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
> swapper/1/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
>  (&(&pThis->Spinlock)->rlock){?.+...}, at: [<f883c7a9>] VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
> {HARDIRQ-ON-W} state was registered at:
>   [<c104b99a>] __lock_acquire+0x509/0x1337
>   [<c104cb3a>] lock_acquire+0x42/0x59
>   [<c13204d5>] _raw_spin_lock+0x25/0x34
>   [<f883c7b3>] VBoxHost_RTSpinlockAcquire+0x1f/0x24 [vboxdrv]
>   [<f883986f>] VBoxHost_RTMpNotificationRegister+0x36/0x11a [vboxdrv]
>   [<f88324f8>] supdrvInitDevExt+0x4f4/0x65c [vboxdrv]
>   [<f8868053>] VBoxDrvLinuxInit+0x53/0xcd [vboxdrv]
>   [<c100106b>] do_one_initcall+0x6b/0x10c
>   [<c1054299>] sys_init_module+0x122f/0x13fa
>   [<c1321990>] sysenter_do_call+0x12/0x36
> irq event stamp: 278096
> hardirqs last  enabled at (278093): [<c1007c3a>] mwait_idle+0x57/0x7c
> hardirqs last disabled at (278094): [<c1320f34>] call_function_interrupt+0x28/0x34
> softirqs last  enabled at (278096): [<c102e3db>] _local_bh_enable+0xd/0xf
> softirqs last disabled at (278095): [<c102e7a8>] irq_enter+0x29/0x4e
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>        CPU0
>        ----
>   lock(&(&pThis->Spinlock)->rlock);
>   <Interrupt>
>     lock(&(&pThis->Spinlock)->rlock);
> 
>  *** DEADLOCK ***

So this is an obvious bug in virtualbox. No need to send us a config.
This bug has nothing to do with RT. It's just that RT can expose bugs in
mainline (or anything added to mainline) much easier.

Funny thing is, this deadlock isn't even possible in RT, so there's
probably many more bugs in the virtualbox driver.

-- Steve

> 
> no locks held by swapper/1/0.
> 
> stack backtrace:
> Pid: 0, comm: swapper/1 Tainted: G           O 3.2.34-2 #1
> Call Trace:
>  [<c131c0bb>] print_usage_bug.part.27+0x1f0/0x1fa
>  [<c104b313>] mark_lock+0x334/0x4b2
>  [<c104a95b>] ? check_usage_backwards+0xce/0xce
>  [<c104b92d>] __lock_acquire+0x49c/0x1337
>  [<c104cb3a>] lock_acquire+0x42/0x59
>  [<f883c7a9>] ? VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
>  [<c1320583>] _raw_spin_lock_irqsave+0x2e/0x3e
>  [<f883c7a9>] ? VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
>  [<f883c7a9>] VBoxHost_RTSpinlockAcquire+0x15/0x24 [vboxdrv]
>  [<f8831d69>] ? supdrvGipMpEventOnline+0x14d/0x14d [vboxdrv]
>  [<f8831c77>] supdrvGipMpEventOnline+0x5b/0x14d [vboxdrv]
>  [<f8831d69>] ? supdrvGipMpEventOnline+0x14d/0x14d [vboxdrv]
>  [<f8831d77>] supdrvGipInitOnCpu+0xe/0x10 [vboxdrv]
>  [<f883b688>] rtmpLinuxWrapper+0x22/0x2d [vboxdrv]
>  [<f883b666>] ? VBoxHost_RTMpCpuId+0xa/0xa [vboxdrv]
>  [<c1051253>] generic_smp_call_function_interrupt+0x6b/0x112
>  [<c1014595>] smp_call_function_interrupt+0x20/0x2e
>  [<c1320f3b>] call_function_interrupt+0x2f/0x34
>  [<c104007b>] ? __run_hrtimer.isra.28+0x33/0x9e
>  [<c1007c42>] ? mwait_idle+0x5f/0x7c
>  [<c10014c2>] cpu_idle+0x4d/0x76
>  [<c14ff2e8>] start_secondary+0x1ab/0x1b2
> vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
> vboxdrv: Successfully loaded version 4.2.4 (interface 0x001a0004).
> vboxpci: pci-stub module not available, cannot detach PCI devices
> vboxpci: IOMMU not found (not compiled)
> --------------- END dmesg output -----------------
> 
> I attached the kernel configuration used for this test.
> 
> Thanks for the support.
> 
> Regards and merry Christmas.
> 
> Mathias
> 

Re: AW: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Thomas Gleixner]

On Fri, 21 Dec 2012, Steven Rostedt wrote:
> On Fri, 2012-12-21 at 08:23 +0000, Koehrer Mathias (ETAS/ESS2) wrote:
> >  *** DEADLOCK ***
> 
> So this is an obvious bug in virtualbox. No need to send us a config.
> This bug has nothing to do with RT. It's just that RT can expose bugs in
> mainline (or anything added to mainline) much easier.
> 
> Funny thing is, this deadlock isn't even possible in RT, so there's
> probably many more bugs in the virtualbox driver.

It's worse on RT, because that's a smp function call interrupt. Which
just cannot work on RT with the lock not being a raw lock.
 
Thanks,

	tglx

Re: AW: VirtualBox 4.2.4 freezes PC using RT_PREEMPT kernel

[Steven Rostedt]

On Fri, 2012-12-21 at 16:05 +0100, Thomas Gleixner wrote:
> On Fri, 21 Dec 2012, Steven Rostedt wrote:
> > On Fri, 2012-12-21 at 08:23 +0000, Koehrer Mathias (ETAS/ESS2) wrote:
> > >  *** DEADLOCK ***
> > 
> > So this is an obvious bug in virtualbox. No need to send us a config.
> > This bug has nothing to do with RT. It's just that RT can expose bugs in
> > mainline (or anything added to mainline) much easier.
> > 
> > Funny thing is, this deadlock isn't even possible in RT, so there's
> > probably many more bugs in the virtualbox driver.
> 
> It's worse on RT, because that's a smp function call interrupt. Which
> just cannot work on RT with the lock not being a raw lock.
>  

Ah, I just figured it was an normal interrupt thread. I was avoiding eye
cancer by not looking at the code. I guess I could have figured it out
with looking at the rest of the lockdep dump, but was too lazy ;-)

-- Steve