Re: [SPDK] bdev/fio_plugin coremask

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4044 bytes --]

On Mon, 2018-07-30 at 22:58 +0300, Andrey Kuzmin wrote:
> I believe my fio threads also affinitize anbd run as requested. What I'm
> concerned with is whether spdk pinned threads are started as requested by the
> core mask, and it looks like they aren't. Not something really concerning as
> fio plugin runs atop fio threads, but still a discrepancy (and likeley worth a
> note in the plugin code/docs).

Making DPDK and fio play together is very tricky. The main problem is that both
DPDK and fio want to be in charge of spawning and affinitizing the threads
(they're both frameworks). There is simply no way to "hook" the creation of a
thread in fio to force it to use an existing one, nor is there a way to force a
thread to become a DPDK thread (there is no equivalent of
spdk_allocate_thread(), for example). Fortunately, all of the DPDK data
structures and API calls we depend on can be called safely from a non-DPDK
thread. Some of them have worse performance in this case (the big one is that
you lose access to the mempool's per-thread cache), but they all at least
function correctly. So the strategy I took here was to spawn DPDK on just one
thread ("the initialization thread"), unaffinitize it, and do nothing on it.
Then do all operations from the fio threads which are spawned according to your
fio configuration file.

When we first implemented this, we did notice some fairly significant
performance hits when pulling bdev_io out of the central pool. However, Jim
added some code that made SPDK's bdev layer manage its own per-SPDK-thread
bdev_io caches a few months ago, so this is no longer a concern. When SPDK
threads become "lightweight" threads that can get rescheduled onto various real
threads at any time, changes like that will become increasingly important anyway
- you want the cache to follow the "lightweight" thread around.

Thanks,
Ben


> 
> Regards,
> Andrey
> >  
> > fio-3.3
> > Starting 2 threads
> > Starting SPDK v18.04 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid133108 ]
> >  
> > But I can see the CPU cores being used are CPU core 28 and 29 per my fio
> > configuration file.
> > 
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> > Sent: Sunday, July 29, 2018 1:47 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] bdev/fio_plugin coremask
> >  
> > Not sure if it's a bogus related to how fio_plugin initializes DPDK
> > environment, but I'm confused by the discrepancy between what I have in the
> > SPDK config file used to run FIO plugin, which is
> > ...
> >   # Users can restrict work items to only run on certain cores by
> >   #  specifying a ReactorMask.  Default is to allow work items to run
> >   #  on core 0.
> >   ReactorMask 0x7
> > ...
> > and what FIO tells me on startup:
> > ...
> > fio-3.8-21-ga8712
> > Starting 1 thread
> > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: fio -c 0x1 --file-prefix=spdk_pid6744 ]
> > EAL: Detected 4 lcore(s)
> > EAL: Multi-process socket /var/run/.spdk_pid6744_unix
> > EAL: Probing VFIO support...
> > ...
> >  
> > If I run bdevperf with the same SPDK config file, I'm quite the contrary
> > getting exactly what I asked for:
> > ...
> > Starting SPDK v18.04.1 / DPDK 18.02.0 initialization...
> > [ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6915 ]
> > EAL: Detected 4 lcore(s)
> > EAL: Multi-process socket /var/run/.spdk_pid6915_unix
> > EAL: Probing VFIO support...
> > ...
> >  
> > Comments will be appreciated.
> >  
> > Regards,
> > Andrey
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> -- 
> Regards,
> Andrey
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk