Re: [SPDK] Tool to monitor disk activity

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2110 bytes --]

On Wed, 2016-10-12 at 04:02 +0000, Naoki Iwakami wrote:
> Hello,
> 
> I am looking for a way to monitor disk activity while I access an SSD via
> SPDK.
> For example, any disk activity does not appear in vmstat or iostat
> while I am running SPDK perf tool.
> I understand it is expected, but I wonder if there is any tool
> available to replace iostat for SPDK.
> Or is there any SPDK API method that helps probing disk activities?
> 

I think this is an important problem to solve as SPDK matures into a solution
used in production deployments, and we've been giving it some thought. Today,
our only available tool is our tracing library (include/spdk/trace.h,
lib/trace), which is an efficient mechanism to add small trace points to the
code. The trace points are placed in a shared memory region that can be mapped
by other tools to do analysis. We're currently investigating either expanding
this (adding well documented tracepoints instead of the ad hoc ones we have in
there now), or changing the format. At some point someone really needs to write
a simplified
iostat-like tool that consumes these tracepoints and displays I/Ops and
bandwidth. We have a very basic example of how this would work (app/iscsi_top)
that displays live connection statistics for the iSCSI target, so maybe that's a
reasonable starting point. It would be awesome if someone from the community
took on the task of formalizing the tracepoints and writing the tool!

We're also working on multi-process support for the NVMe driver, which means
multiple processes
will be able to create I/O queues for the same NVMe device. This could
potentially allow tools to attach to an active SPDK process and send commands
directly to the SSD to query
performance metrics, if the SSD provides them. I think our long term vision for
this feature is to enable support for configuration and management of SPDK-owned 
NVMe SSDs with the standard nvme-cli tool, but we're quite a ways off yet.

We're very open to ideas and feedback in this area if anyone has anything to say
on the subject!

> Thank you,