[SPDK] Re: A question for bdev_io cache

[SPDK] Re: A question for bdev_io cache

[Yijing Wang]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

Hi Jim, thanks for your information, now I understand why we use spdk 
bdev owner cache.

Thanks!

Yijing.


在 2019/10/9 下午12:06, Harris, James R 写道:
> Hi Yijing,
>
> We used to use a DPDK rte_mempool for the spdk_bdev_io structures.  Here's the commit where we changed it:
>
> https://github.com/spdk/spdk/commit/534d9c20020018aa2eac0a113987f6ba04855b5a
>
> The primary problem we had with the rte_mempool cache is that it only works on DPDK threads.  So applications that use the SPDK bdev layer without using DPDK threads would incur massive contention on the rte_mempool global lock.  This was especially evident with the SPDK bdev fio_plugin, which was specifically designed to show how to use the SPDK bdev layer without using DPDK threads (or the rest of the SPDK event framework).
>
> -Jim
>
> 	
> On 10/8/19, 7:09 PM, "Yijing Wang" <wangyijing0307(a)gmail.com> wrote:
>
>      Hi James,
>      
>            I'm a spdk developer from china, I'm sorry to disturb you,
>      recently I read the latest spdk bdev io code, and found that spdk
>      prepopulate per-thread bdev_io cache in bdev module.
>      
>      As we knew, dpdk rte_mempool_create has a parameter cache_size which
>      allow users to cache some objects in local cpu,  my question is why you
>      post the patch to create bdev io cache
>      
>      in bdev own list, but not use default dpdk rte_mempool cache mechnasim？
>      The reason is dpdk rte_mempool cache cannot prevent a bdev device in a
>      thread from starvation? Since dpdk
>      
>      rte_mempool cache is preserved for all users in the same cpu thread.
>      
>      
>      Thanks!
>      
>      Yijing.
>      
>

[SPDK] Re: A question for bdev_io cache

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]

Hi Yijing,

We used to use a DPDK rte_mempool for the spdk_bdev_io structures.  Here's the commit where we changed it:

https://github.com/spdk/spdk/commit/534d9c20020018aa2eac0a113987f6ba04855b5a

The primary problem we had with the rte_mempool cache is that it only works on DPDK threads.  So applications that use the SPDK bdev layer without using DPDK threads would incur massive contention on the rte_mempool global lock.  This was especially evident with the SPDK bdev fio_plugin, which was specifically designed to show how to use the SPDK bdev layer without using DPDK threads (or the rest of the SPDK event framework).

-Jim

	
On 10/8/19, 7:09 PM, "Yijing Wang" <wangyijing0307(a)gmail.com> wrote:

    Hi James,
    
          I'm a spdk developer from china, I'm sorry to disturb you, 
    recently I read the latest spdk bdev io code, and found that spdk 
    prepopulate per-thread bdev_io cache in bdev module.
    
    As we knew, dpdk rte_mempool_create has a parameter cache_size which 
    allow users to cache some objects in local cpu,  my question is why you 
    post the patch to create bdev io cache
    
    in bdev own list, but not use default dpdk rte_mempool cache mechnasim？ 
    The reason is dpdk rte_mempool cache cannot prevent a bdev device in a 
    thread from starvation? Since dpdk
    
    rte_mempool cache is preserved for all users in the same cpu thread.
    
    
    Thanks!
    
    Yijing.