[Question] A novel case happened when using mempool allocate memory.

[Question] A novel case happened when using mempool allocate memory.

[zhong jiang]

Hi,  Everyone

 I ran across the following novel case similar to memory leak in linux-4.1 stable when allocating
 memory object by kmem_cache_alloc.   it rarely can be reproduced.

I create a specific  mempool with 24k size based on the slab.  it can not be merged with
other kmem cache.  I  record the allocation and free usage by atomic_add/sub.    After a while,
I watch the specific slab consume most of total memory.   After halting the code execution.
The counter of allocation and free is equal.  Therefore,  I am sure that module have released
all meory resource.  but the statistic of specific slab is very high but stable by checking /proc/slabinfo.

but It is strange that the specific slab will free get back all memory when unregister the module.
I got the following information from specific slab data structure when halt the module execution.


kmem_cache_node                                                          kmem_cache

nr_partial = 1,                                                             min_partial = 7
partial = {                                                                    cpu_partial = 2
        next = 0xffff7c00085cae20                             object_size = 24576
        prev = 0xffff7c00085cae20
},

nr_slabs = {
    counter = 365610
 },

total_objects = {
 counter = 365610
},

full = {
      next =  0xffff8013e44f75f0,
     prev =  0xffff8013e44f75f0
},

>From the above restricted information , we can know that the node full list is empty.  and partial list only
have a  slab.   A slab contain a object.  I think that most of slab stay in the cpu_partial
list even though it seems to be impossible theoretically.  because I come to the conclusion based on the case
that slab take up the memory will be release when unregister the moudle.

but I check the code(mm/slub.c) carefully . I can not find any clue to prove my assumption.
I will be appreciate if anyone have any idea about the case. 


Thanks
zhong jiang

Re: [Question] A novel case happened when using mempool allocate memory.

[Matthew Wilcox]

On Wed, Aug 01, 2018 at 11:31:15PM +0800, zhong jiang wrote:
> Hi,  Everyone
> 
>  I ran across the following novel case similar to memory leak in linux-4.1 stable when allocating
>  memory object by kmem_cache_alloc.   it rarely can be reproduced.
> 
> I create a specific  mempool with 24k size based on the slab.  it can not be merged with
> other kmem cache.  I  record the allocation and free usage by atomic_add/sub.    After a while,
> I watch the specific slab consume most of total memory.   After halting the code execution.
> The counter of allocation and free is equal.  Therefore,  I am sure that module have released
> all meory resource.  but the statistic of specific slab is very high but stable by checking /proc/slabinfo.

Please post the code.

Re: [Question] A novel case happened when using mempool allocate memory.

[zhong jiang]

On 2018/8/1 23:37, Matthew Wilcox wrote:
> On Wed, Aug 01, 2018 at 11:31:15PM +0800, zhong jiang wrote:
>> Hi,  Everyone
>>
>>  I ran across the following novel case similar to memory leak in linux-4.1 stable when allocating
>>  memory object by kmem_cache_alloc.   it rarely can be reproduced.
>>
>> I create a specific  mempool with 24k size based on the slab.  it can not be merged with
>> other kmem cache.  I  record the allocation and free usage by atomic_add/sub.    After a while,
>> I watch the specific slab consume most of total memory.   After halting the code execution.
>> The counter of allocation and free is equal.  Therefore,  I am sure that module have released
>> all meory resource.  but the statistic of specific slab is very high but stable by checking /proc/slabinfo.
> Please post the code.
>
> .
>

when module is loaded. we create the specific mempool. The code flow is as follows.

mem_pool_create() {

slab_cache = kmem_cache_create(name, item_size, 0, 0 , NULL);

mempoll_create(min_pool_size, mempool_alloc_slab, mempool_free_slab, slab_cache);   //min_pool_size is assigned to 1024
atomic_set(pool->statistics, 0);
}

we allocate memory from specific mempool , The code flow is as follows.

mem_alloc() {
mempool_alloc(pool, gfp_flags);

atomic_inc(pool->statistics);
}

we release memory to specific mempool . The code flow is as follows.
mem_free() {
mempool_free(object_ptr, pool);

atomic_dec(pool->statistics);
}


when we unregister the module,  the memory has been taken up will get back the system.
the code flow is as follows.

mem_pool_destroy() {
 mempool_destroy(pool);
kmem_cache_destroy(slab_cache);
}

>From the above information.  I assume the specific kmem_cache will not take up overmuch memory
when halting the execution and pool->statistics is equal to 0.

I have no idea about the issue. 

Thanks
zhong jiang

Re: [Question] A novel case happened when using mempool allocate memory.

[Matthew Wilcox]

On Thu, Aug 02, 2018 at 02:22:03PM +0800, zhong jiang wrote:
> On 2018/8/1 23:37, Matthew Wilcox wrote:
> > Please post the code.
> 
> when module is loaded. we create the specific mempool. The code flow is as follows.

I actually meant "post the code you are testing", not "write out some
pseudocode".

Re: [Question] A novel case happened when using mempool allocate memory.

[zhong jiang]

On 2018/8/2 21:31, Matthew Wilcox wrote:
> On Thu, Aug 02, 2018 at 02:22:03PM +0800, zhong jiang wrote:
>> On 2018/8/1 23:37, Matthew Wilcox wrote:
>>> Please post the code.
>> when module is loaded. we create the specific mempool. The code flow is as follows.
> I actually meant "post the code you are testing", not "write out some
> pseudocode".
>
> .
>
The source code is as follow about mempool utility.

**
* @brief a??c' c??c+->>a??e!oao?a,? @smio_mem_type_tc??a(R)?a1?e!oao?a,?e?'
*/
static smio_mem_mng_t g_smio_mem[] =
{
{
.name = "MEDIA_INFO",
.min_pool_size = 128,
.item_size = sizeof(smio_media_info_t),
.slab_cache = NULL,
},
{
.name = "DSW_IO_REQ",
.min_pool_size = 1024,
.item_size = sizeof(dsw_io_req_t),
.slab_cache = NULL,
},
{
.name = "DSW_IO_PAGE",
.min_pool_size = 1024,
.item_size = sizeof(dsw_page_t) * DSW_MAX_PAGE_PER_REQ,
.slab_cache = NULL,
},
{
.name = "32_ARRAY",
.min_pool_size = 1024,
.item_size = sizeof(void *) * 32,
.slab_cache = NULL,
},
{
.name = "SCSI_SENSE_BUF",
.min_pool_size = 1024,
.item_size = sizeof(char) * SCSI_SENSE_BUFFERSIZE,
.slab_cache = NULL,
},
};

/**
* @brief c?3e?.ae??ae?(R)c+->>a??a??a-?
*
* @param id c?3e?.e??ae?!a??ID
* @param type c?3e?.a??a-?c??c+->>a??
*
* @return ae??a??e??a??a??a-?a??c??e|?a??a??;a?+-e'JPYe??a??NULL
*/
void *smio_mem_alloc(smio_module_id_t id, smio_mem_type_t type)
{
void *m = NULL;
smio_mem_mng_t *pool_mng = NULL;
SMIO_ASSERT_RETURN(id < SMIO_MOD_ID_BUTT, NULL);
SMIO_ASSERT_RETURN(type < SMIO_MEM_TYPE_BUTT, NULL);

pool_mng = &g_smio_mem[type];

SMIO_LOG_DEBUG("alloc %s, size: %d\n", pool_mng->name, pool_mng->item_size);

m = mempool_alloc(pool_mng->pool, GFP_KERNEL);
if (NULL == m)
{
return NULL;
}

memset(m, 0, pool_mng->item_size);

atomic_inc(&pool_mng->statistics[id]);

return m;
}
EXPORT_SYMBOL(smio_mem_alloc);


/**
* @brief e??ae? 3/4 a??a-?a??
*
* @param id c?3e?.e??c??ae?!a??ID
* @param type a??a-?a??c??c+->>a??
* @param m a??a??c??e|?a??a??
*/
void smio_mem_free(smio_module_id_t id, smio_mem_type_t type, void *m)
{
smio_mem_mng_t *pool_mng = NULL;
SMIO_ASSERT(NULL != m);
SMIO_ASSERT(id < SMIO_MOD_ID_BUTT);
SMIO_ASSERT(type < SMIO_MEM_TYPE_BUTT);

pool_mng = &g_smio_mem[type];

mempool_free(m, pool_mng->pool);

atomic_dec(&pool_mng->statistics[id]);
}
EXPORT_SYMBOL(smio_mem_free);


/**
* @brief a??a>>oc(R)!c??a??a??ae+- 
*
* @param pool_mng a??a-?c+->>a??c(R)!c??c>>?ae??
*
* @return ae??a??e??a??@SMIO_OK;a?+-e'JPYe??a??@SMIO_ERR
*/
static int smio_mem_pool_create(smio_mem_mng_t *pool_mng)
{
int i;
SMIO_ASSERT_RETURN(NULL != pool_mng, SMIO_ERR);

pool_mng->slab_cache = kmem_cache_create(pool_mng->name,
pool_mng->item_size, 0, 0, NULL);

if (SMIO_IS_ERR_OR_NULL(pool_mng->slab_cache))
{
SMIO_LOG_ERR("kmem_cache_create for %s failed\n", pool_mng->name);
return SMIO_ERR;
}
pool_mng->pool = mempool_create(pool_mng->min_pool_size, mempool_alloc_slab,
mempool_free_slab, pool_mng->slab_cache);
if (NULL == pool_mng->pool)
{
SMIO_LOG_ERR("pool create for %s failed\n", pool_mng->name);
kmem_cache_destroy(pool_mng->slab_cache);
return SMIO_ERR;
}

for (i = 0; i < SMIO_MOD_ID_BUTT; i++)
{
atomic_set(&pool_mng->statistics[i], 0);
}

return SMIO_OK;
}


/**
* @brief ae,?e??a??a-?ae+- 
*
* @param pool_mng ae??e|?ae,?e??c??a??a-?ae+- 
*/
void smio_mem_pool_destroy(smio_mem_mng_t *pool_mng)
{
SMIO_ASSERT(NULL != pool_mng);

if (NULL != pool_mng->pool)
{
mempool_destroy(pool_mng->pool);
pool_mng->pool = NULL;
}

if (NULL != pool_mng->slab_cache)
{
kmem_cache_destroy(pool_mng->slab_cache);
pool_mng->slab_cache = NULL;
}
}


/**
* @brief a??a-?c(R)!c??a??a??a??a??a??
*
* @return ae??a??e??a??@SMIO_OK;a?+-e'JPYe??a??@SMIO_ERR
*/
int smio_mem_init(void)
{
int i;
int pool_num = (int) SMIO_ARRAY_SIZE(g_smio_mem);
int ret = SMIO_OK;
bool free = SMIO_FALSE;

for (i = 0; i < pool_num; i++)
{
SMIO_LOG_INFO("memory of %s initialize, min_pool_size: %d, item_size: %d\n",
g_smio_mem[i].name, g_smio_mem[i].min_pool_size, g_smio_mem[i].item_size);
if (SMIO_OK != smio_mem_pool_create(&g_smio_mem[i]))
{
SMIO_LOG_ERR("memory of %s initialize failed\n", g_smio_mem[i].name);
ret = SMIO_ERR;
free = SMIO_TRUE;
break;
}
}

/* clean if smio_mem_pool_create failed*/
while ((SMIO_TRUE == free) && (--i >= 0))
{
smio_mem_pool_destroy(&g_smio_mem[i]);
}

return ret;
}


/**
* @brief a??a-?c(R)!c??ae?!a??ae,?e??e??a?o
*/
void smio_mem_exit(void)
{
int i;
int pool_num = (int) SMIO_ARRAY_SIZE(g_smio_mem);

for (i = 0; i < pool_num; i++)
{
smio_mem_pool_destroy(&g_smio_mem[i]);
}
}