kmem_cache_destroy issue with RT12 patch on v3.0.3 (perhaps related to thread 3.0.4 + rt12: deadlock?)

kmem_cache_destroy issue with RT12 patch on v3.0.3 (perhaps related to thread 3.0.4 + rt12: deadlock?)

[PRINGLE Chris]

Dear All,

I've just upgraded to v3.0.3 of the kernel and the latest set of corresponding RT patches (rt12) and have run into a kernel OOPS/hang. This didn't happen on earlier kernels (2.6.33.7 was the last I tried), and doesn't happen without the RT patch.

The following code causes the a kernel OOPS at the point where I call kmem_cache_destroy:

struct test_struct
{
    unsigned int test[4096];
};

#define NUM_TESTS 1000

bool test_func(void)
{
    struct kmem_cache* testcache;
    struct test_struct* structs[NUM_TESTS];
    int i;

    printk(KERN_ERR "Creating\n");
    testcache= KMEM_CACHE(test_struct, 0);
    if (testcache == NULL)
    {
        printk(KERN_ERR "ERROR: %s: Failed to allocate lookaside cache \n", __func__);
        return false;
    }

    printk(KERN_ERR "Allocating\n");
    for (i= 0; i < NUM_TESTS; i++)
    {
        structs[i]= kmem_cache_alloc(testcache, GFP_KERNEL);
        if (structs[i] == NULL)
        {
            printk(KERN_ERR "ERROR: Failed to allocate\n");
        }
    }

    printk(KERN_ERR "Freeing\n");
    for (i= 0; i < NUM_TESTS; i++)
    {
        if (structs[i])
            kmem_cache_free(testcache, structs[i]);
    }

    printk(KERN_ERR "Destroying\n");
    kmem_cache_destroy(testcache);
}

Stack trace (from gdb) is as follows:
#23 0xc137ee53 in rt_spin_lock_slowlock (lock=0xcf802be0) at kernel/rtmutex.c:781
#24 0xc137f120 in rt_spin_lock_fastlock (lock=0xcedf00f1) at kernel/rtmutex.c:650
#25 rt_spin_lock (lock=0xcedf00f1) at kernel/rtmutex.c:815
#26 0xc1099818 in __local_lock_irq (cachep=0xcf400200, objp=<value optimised out>) at include/linux/locallock.h:89
#27 __local_lock_irqsave (cachep=0xcf400200, objp=<value optimised out>) at include/linux/locallock.h:117
#28 kmem_cache_free (cachep=0xcf400200, objp=<value optimised out>) at mm/slab.c:3940
#29 0xc10999b7 in slab_destroy (cachep=0xcec0bf00, slabp=<value optimised out>, delayed=<value optimised out>) at mm/slab.c:2100
#30 0xc1099b34 in free_block (cachep=0xcec0bf00, objpp=<value optimised out>, nr_objects=<value optimised out>, node=0) at mm/slab.c:3674
#31 0xc1099f6e in __do_drain (arg=0xcec0bf00, cpu=<value optimised out>) at mm/slab.c:2606
#32 do_drain (arg=0xcec0bf00, cpu=<value optimised out>) at mm/slab.c:2622
#33 0xc1098bde in slab_on_each_cpu (func=0xc1099f10 <do_drain>, arg=0xcec0bf00) at mm/slab.c:737
#34 0xc109a296 in drain_cpu_caches (cachep=0xcec0bf00) at mm/slab.c:2634
#35 __cache_shrink (cachep=0xcec0bf00) at mm/slab.c:2696
#36 0xc109a46c in kmem_cache_destroy (cachep=0xcec0bf00) at mm/slab.c:2760

The code is running on an SMP i686 kernel with full pre-emption enabled. The kernel is running on an i686 VMWare virtual machine.

If I add SLAB_RED_ZONE | SLAB_DEBUG_OBJECTS to KMEM_CACHE flags the problem doesn't appear to show itself.

Any ideas?
Cheers,
Chris

____________________________

Miranda Technologies Limited
Registered in England and Wales CN 02017053
Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kmem_cache_destroy issue with RT12 patch on v3.0.3 (perhaps related to thread 3.0.4 + rt12: deadlock?)

[Peter Zijlstra]

On Fri, 2011-09-02 at 12:02 +0100, PRINGLE Chris wrote:

> The following code causes the a kernel OOPS at the point where I call
> kmem_cache_destroy:
> 


What does that oops look like?

RE: kmem_cache_destroy issue with RT12 patch on v3.0.3 (perhaps related to thread 3.0.4 + rt12: deadlock?)

[PRINGLE Chris]

Dear All,

We've just tried v3.0.34-rt55 on a PowerPC and we've got the same results that we had on the i686 nearly a year ago. The OOPS is as follows:

------------[ cut here ]------------
kernel BUG at kernel/rtmutex.c:725!
Oops: Exception in kernel mode, sig: 5 [#1]
PREEMPT SMP NR_CPUS=8 CPC-3801
Modules linked in: cpc(-) [last unloaded: cpc]
NIP: c04e5058 LR: c04e503c CTR: 00000000
REGS: e8043c00 TRAP: 0700   Not tainted  (3.0.34-rt55)
MSR: 00029002 <EE,ME,CE>  CR: 82000842  XER: 20000000
TASK = e80207f0[906] 'rmmod' THREAD: e8042000 CPU: 3
GPR00: 00000001 e8043cb0 e80207f0 00000000 e80207f0 00000000 00000001 00000000 
GPR08: e80207f0 e80207f0 e80207f0 e80207f1 00000001 1001a534 bff65560 00000000 
GPR16: 00000004 00000000 00000020 00000000 00000004 00000000 00000020 00000000 
GPR24: c065d234 e9b02628 00100100 c065b348 e9f8cf20 c062dc24 e80207f0 c0cfdc24 
NIP [c04e5058] rt_spin_lock_slowlock+0x80/0x278
LR [c04e503c] rt_spin_lock_slowlock+0x64/0x278
Call Trace:
[e8043cb0] [c04e503c] rt_spin_lock_slowlock+0x64/0x278 (unreliable)
[e8043d20] [c00daa58] kmem_cache_free+0x74/0x11c
[e8043d40] [c00dad3c] free_block+0x1a4/0x1b0
[e8043d70] [c00dafac] do_drain+0x94/0x124
[e8043db0] [c00d9ef8] slab_on_each_cpu+0x48/0x90
[e8043dd0] [c00db3b4] __cache_shrink+0x28/0xc8
[e8043de0] [c00db664] kmem_cache_destroy+0x74/0x118
[e8043df0] [f3041894] cpc_command_buffer_term+0x40/0xd4 [cpc]
[e8043e00] [f3042f34] cpc_command_buffer_queue_term+0x38/0x68 [cpc]
[e8043e10] [f304973c] cpc_remove+0x38/0x1bc [cpc]
[e8043e30] [c028b7e4] pci_device_remove+0x38/0x74
[e8043e40] [c02c8a88] __device_release_driver+0x6c/0xd0
[e8043e50] [c02c8bd0] driver_detach+0xe4/0xe8
[e8043e70] [c02c7950] bus_remove_driver+0x9c/0x114
[e8043e90] [c02c94f4] driver_unregister+0x60/0x8c
[e8043ea0] [c028bb0c] pci_unregister_driver+0x58/0xf0
[e8043ec0] [f30496f4] cpc_exit+0x18/0x28 [cpc]
[e8043ed0] [c00832f4] sys_delete_module+0x198/0x238
[e8043f40] [c000f10c] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xfe9aa2c
   LR = 0x1000198c
Instruction dump:
7fe3fb78 7fc4f378 38a00000 38c00001 4bb9725d 2f830000 409e0174 801f0010 
5400003c 7fc00278 7c000034 5400d97e <0f000000> 3bbe0500 3b810008 7fa3eb78 
---[ end trace 0000000000000002 ]---
note: rmmod[906] exited with preempt_count 1


Does anyone have any ideas what this might be? The problem only occurs when calling kmem_cache_destroy after a large volume of objects have passed through it.

Cheers,
Chris




-----Original Message-----
From: Peter Zijlstra [mailto:peterz@infradead.org] 
Sent: 06 September 2011 12:59
To: PRINGLE Chris
Cc: linux-rt-users@vger.kernel.org
Subject: Re: kmem_cache_destroy issue with RT12 patch on v3.0.3 (perhaps related to thread 3.0.4 + rt12: deadlock?)

On Fri, 2011-09-02 at 12:02 +0100, PRINGLE Chris wrote:

> The following code causes the a kernel OOPS at the point where I call
> kmem_cache_destroy:
> 


What does that oops look like?