[SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

Hi All,

I read from the SPDK doc "NVMe Driver Design -- Scaling Performance" (here
<http://www.spdk.io/doc/nvme.html#nvme_design>), which saids:

" For example, if a device claims to be capable of 450,000 I/O per second
at queue depth 128, in practice it does not matter if the driver is using 4
queue pairs each with queue depth 32, or a single queue pair with queue
depth 128."

Does this consider the queuing latency? I am guessing the latency in the
two cases will be different ( in qp/qd = 4/32 and in qp/qd = 1/128). In the
4 threads case, the latency will be 1/4 of the 1 thread case. Do I get it
right?

If so, then I got confused as the document also says:

"In order to take full advantage of this scaling, applications should
consider organizing their internal data structures such that data is
assigned exclusively to a single thread."

Please correct me if I get it wrong. I understand that if the dedicate I/O
thread has the total ownership of the I/O data structures, there is no lock
contention to slow down the I/O. I believe that BlobFS is also designed in
this philosophy in that only one thread is doing I/O.

But considering the RocksDB case, if the shared data structure has already
been largely taken care of by the RocksDB logic via locking (which is
inevitable anyway), the I/O requests each RocksDB thread sends to the
BlobFS could also has its own queue pair to do I/O. More I/O threads means
shorter queue depth and smaller queuing delay.

Even if there is some FS metadata operations that may require some locking,
but I would guest such metadata operation takes only a small portion.

Therefore, is it a viable idea to have more I/O threads in the BlobFS to
serve the multi-threaded RocksDB for a smaller delay? What will be the
pitfalls, or challenges?

Any thoughts/comments are appreciated. Thank you very much!

Best!
-Fenggang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2208 bytes --]

Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 3278 bytes --]

Hi Fenggang,

The max IOPs number is per-device – not per-queue.  The observed latency for each I/O - from submission to completion - will be the same whether the 128 I/O are submitted on one queue or across four queues.  Spreading the I/O across four queues instead of one just means that the device will process ¼ the rate of I/O from each of the four queues compared to if it was submitted on a single queue.

For BlobFS, spreading the I/O across multiple NVMe queues would not normally help with latency.  There are NVMe features such as Weighted Round Robin (WRR), which provide different priorities to different queues.  With WRR, multiple NVMe queues could be used to separate high priority I/O (i.e. WAL writes) from lower priority I/O (i.e. background compaction I/O).  Most NVMe devices today do not support WRR however and even then it’s still questionable whether WRR alone would be sufficient or if additional software queuing would be required.

Thanks,
-Jim




From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Fenggang Wu <fenggang(a)cs.umn.edu>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, January 31, 2018 at 10:49 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Cc: "wuxx0835(a)umn.edu" <wuxx0835(a)umn.edu>
Subject: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

Hi All,

I read from the SPDK doc "NVMe Driver Design -- Scaling Performance" (here<http://www.spdk.io/doc/nvme.html#nvme_design>), which saids:

" For example, if a device claims to be capable of 450,000 I/O per second at queue depth 128, in practice it does not matter if the driver is using 4 queue pairs each with queue depth 32, or a single queue pair with queue depth 128."

Does this consider the queuing latency? I am guessing the latency in the two cases will be different ( in qp/qd = 4/32 and in qp/qd = 1/128). In the 4 threads case, the latency will be 1/4 of the 1 thread case. Do I get it right?

If so, then I got confused as the document also says:

"In order to take full advantage of this scaling, applications should consider organizing their internal data structures such that data is assigned exclusively to a single thread."

Please correct me if I get it wrong. I understand that if the dedicate I/O thread has the total ownership of the I/O data structures, there is no lock contention to slow down the I/O. I believe that BlobFS is also designed in this philosophy in that only one thread is doing I/O.

But considering the RocksDB case, if the shared data structure has already been largely taken care of by the RocksDB logic via locking (which is inevitable anyway), the I/O requests each RocksDB thread sends to the BlobFS could also has its own queue pair to do I/O. More I/O threads means shorter queue depth and smaller queuing delay.

Even if there is some FS metadata operations that may require some locking, but I would guest such metadata operation takes only a small portion.

Therefore, is it a viable idea to have more I/O threads in the BlobFS to serve the multi-threaded RocksDB for a smaller delay? What will be the pitfalls, or challenges?

Any thoughts/comments are appreciated. Thank you very much!

Best!
-Fenggang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8550 bytes --]

Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4738 bytes --]

On Wed, 2018-01-31 at 17:49 +0000, Fenggang Wu wrote:
> Hi All, 
> 
> I read from the SPDK doc "NVMe Driver Design -- Scaling Performance" (here),
> which saids:
> 
> " For example, if a device claims to be capable of 450,000 I/O per second at
> queue depth 128, in practice it does not matter if the driver is using 4 queue
> pairs each with queue depth 32, or a single queue pair with queue depth 128."
> 
> Does this consider the queuing latency? I am guessing the latency in the two
> cases will be different ( in qp/qd = 4/32 and in qp/qd = 1/128). In the 4
> threads case, the latency will be 1/4 of the 1 thread case. Do I get it right?

Officially, it is entirely up to the internal design of the device. But for the
NVMe devices I've encountered on the market today you can use as a mental model
a single thread inside the SSD processing incoming messages that correspond to
doorbell writes. It simply takes the doorbell write message and does simple math
 to calculate where the command is located in host memory, and then issues a DMA
to pull it into device local memory. It doesn't matter which queue the I/O is on
- the math is the same. So no, the latency of 1 queue pair at 128 queue depth is
the same as 4 queue pairs at 32 queue depth.

> If so, then I got confused as the document also says:
> 
> "In order to take full advantage of this scaling, applications should consider
> organizing their internal data structures such that data is assigned
> exclusively to a single thread."
> 
> Please correct me if I get it wrong. I understand that if the dedicate I/O
> thread has the total ownership of the I/O data structures, there is no lock
> contention to slow down the I/O. I believe that BlobFS is also designed in
> this philosophy in that only one thread is doing I/O. 
> 
> But considering the RocksDB case, if the shared data structure has already
> been largely taken care of by the RocksDB logic via locking (which is
> inevitable anyway), the I/O requests each RocksDB thread sends to the BlobFS
> could also has its own queue pair to do I/O. More I/O threads means shorter
> queue depth and smaller queuing delay. 

> Even if there is some FS metadata operations that may require some locking,
> but I would guest such metadata operation takes only a small portion.
> 
> Therefore, is it a viable idea to have more I/O threads in the BlobFS to serve
> the multi-threaded RocksDB for a smaller delay? What will be the pitfalls, or
> challenges?

You're right that RocksDB has already worked out all of its internal data
sharing using locks. It then uses a thread pool to issue simultaneous blocking
I/O requests to the filesystem. That's where the SPDK RocksDB backend
intercepts. As you suspect, the filesystem itself (BlobFS, in this case) has
shared data structures that must be coordinated for some operations (creating
and deleting files, resizing files, etc. - but not regular read/write). That's a
small part of the reason why we elected, in our first attempt at writing a
RocksDB backend, to route all I/O from each thread in the thread pool to a
single thread doing asynchronous I/O.

The main reason we route all I/O to a single thread, however, is to minimize CPU
usage. RocksDB makes blocking calls on all threads in the thread pool. We could
implement that in SPDK by spinning in a tight loop, polling for the I/O to
complete. But that means every thread in the RocksDB thread pool would be
burning the full core. Instead, we send all I/O to a single thread that is
polling for completions, and put the threads in the pool to sleep on a
semaphore. When an I/O completes, we send a message back to the originating
thread and kick the semaphore to wake it up. This introduces some latency (but
the rest of SPDK is more than fast enough to compensate for that), but it saves
a lot of CPU usage.

In an ideal world, we'd be integrating with a fully asynchronous K/V database,
where the user could call Put() or Get() and have it return immediately and call
a callback when the data was actually inserted. But that's just not how RocksDB
works today. Even the background thread pool doing compaction is designed to do
blocking operations. It would integrate with SPDK much better if it instead had
a smaller set of threads each doing asynchronous compaction operations on a
whole set of files at once. Changing RocksDB in this way is a huge lift, but
would be an impressive project.

> 
> 
> 
> Any thoughts/comments are appreciated. Thank you very much!
> 
> Best!
> -Fenggang
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 6215 bytes --]

Hi Ben,

Thank you very much!

Fenggang WU(吴凤刚)
Ph.D. Student
Department of Computer Science and Engineering <http://www.cs.umn.edu/>
College of Science and Engineering <http://cse.umn.edu/>
University of Minnesota, Twin Cities <http://www.umn.edu>
Email: wuxx0835(a)umn.edu
Homepage: http://www.cs.umn.edu/~fenggang

On Wed, Jan 31, 2018 at 12:22 PM, Walker, Benjamin <
benjamin.walker(a)intel.com> wrote:

> On Wed, 2018-01-31 at 17:49 +0000, Fenggang Wu wrote:
> > Hi All,
> >
> > I read from the SPDK doc "NVMe Driver Design -- Scaling Performance"
> (here),
> > which saids:
> >
> > " For example, if a device claims to be capable of 450,000 I/O per
> second at
> > queue depth 128, in practice it does not matter if the driver is using 4
> queue
> > pairs each with queue depth 32, or a single queue pair with queue depth
> 128."
> >
> > Does this consider the queuing latency? I am guessing the latency in the
> two
> > cases will be different ( in qp/qd = 4/32 and in qp/qd = 1/128). In the 4
> > threads case, the latency will be 1/4 of the 1 thread case. Do I get it
> right?
>
> Officially, it is entirely up to the internal design of the device. But
> for the
> NVMe devices I've encountered on the market today you can use as a mental
> model
> a single thread inside the SSD processing incoming messages that
> correspond to
> doorbell writes. It simply takes the doorbell write message and does
> simple math
>  to calculate where the command is located in host memory, and then issues
> a DMA
> to pull it into device local memory. It doesn't matter which queue the I/O
> is on
> - the math is the same. So no, the latency of 1 queue pair at 128 queue
> depth is
> the same as 4 queue pairs at 32 queue depth.
>
> > If so, then I got confused as the document also says:
> >
> > "In order to take full advantage of this scaling, applications should
> consider
> > organizing their internal data structures such that data is assigned
> > exclusively to a single thread."
> >
> > Please correct me if I get it wrong. I understand that if the dedicate
> I/O
> > thread has the total ownership of the I/O data structures, there is no
> lock
> > contention to slow down the I/O. I believe that BlobFS is also designed
> in
> > this philosophy in that only one thread is doing I/O.
> >
> > But considering the RocksDB case, if the shared data structure has
> already
> > been largely taken care of by the RocksDB logic via locking (which is
> > inevitable anyway), the I/O requests each RocksDB thread sends to the
> BlobFS
> > could also has its own queue pair to do I/O. More I/O threads means
> shorter
> > queue depth and smaller queuing delay.
>
> > Even if there is some FS metadata operations that may require some
> locking,
> > but I would guest such metadata operation takes only a small portion.
> >
> > Therefore, is it a viable idea to have more I/O threads in the BlobFS to
> serve
> > the multi-threaded RocksDB for a smaller delay? What will be the
> pitfalls, or
> > challenges?
>
> You're right that RocksDB has already worked out all of its internal data
> sharing using locks. It then uses a thread pool to issue simultaneous
> blocking
> I/O requests to the filesystem. That's where the SPDK RocksDB backend
> intercepts. As you suspect, the filesystem itself (BlobFS, in this case)
> has
> shared data structures that must be coordinated for some operations
> (creating
> and deleting files, resizing files, etc. - but not regular read/write).
> That's a
> small part of the reason why we elected, in our first attempt at writing a
> RocksDB backend, to route all I/O from each thread in the thread pool to a
> single thread doing asynchronous I/O.
>
> The main reason we route all I/O to a single thread, however, is to
> minimize CPU
> usage. RocksDB makes blocking calls on all threads in the thread pool. We
> could
> implement that in SPDK by spinning in a tight loop, polling for the I/O to
> complete. But that means every thread in the RocksDB thread pool would be
> burning the full core. Instead, we send all I/O to a single thread that is
> polling for completions, and put the threads in the pool to sleep on a
> semaphore. When an I/O completes, we send a message back to the originating
> thread and kick the semaphore to wake it up. This introduces some latency
> (but
> the rest of SPDK is more than fast enough to compensate for that), but it
> saves
> a lot of CPU usage.
>

Yeah, get it. It make perfect sense to me that BlobFS concentrate the I/Os
to one thread to minimized the busy waiting.



>
> In an ideal world, we'd be integrating with a fully asynchronous K/V
> database,
> where the user could call Put() or Get() and have it return immediately
> and call
> a callback when the data was actually inserted. But that's just not how
> RocksDB
> works today. Even the background thread pool doing compaction is designed
> to do
> blocking operations. It would integrate with SPDK much better if it
> instead had
> a smaller set of threads each doing asynchronous compaction operations on a
> whole set of files at once. Changing RocksDB in this way is a huge lift,
> but
> would be an impressive project.
>
>
Right, if RocksDB could do async compaction, the more parallelism of the
SSD can be exploit using a small number of thread. Or equivalently, RocksDB
can spawn enough blocking compaction threads, in a sense to keep the SSD
busy. However, tradeoff is that there will be more thread managing
overhead.

Still, instead of altering RocksDB, which is complicated, it's also
possible to start from other parallel LSM equivalence such as HyperLevelDB
<http://hyperdex.org/performance/leveldb/>.



> >
> >
> >
> > Any thoughts/comments are appreciated. Thank you very much!
> >
> > Best!
> > -Fenggang
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9241 bytes --]

Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 4470 bytes --]

Hi Jim,

I also have some quick followup question about WRR inline below.

Thank you very much!
-Fenggang

Fenggang WU(吴凤刚)
Ph.D. Student
Department of Computer Science and Engineering <http://www.cs.umn.edu/>
College of Science and Engineering <http://cse.umn.edu/>
University of Minnesota, Twin Cities <http://www.umn.edu>
Email: wuxx0835(a)umn.edu
Homepage: http://www.cs.umn.edu/~fenggang

On Wed, Jan 31, 2018 at 12:09 PM, Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Fenggang,
>
>
>
> The max IOPs number is per-device – not per-queue.  The observed latency
> for each I/O - from submission to completion - will be the same whether the
> 128 I/O are submitted on one queue or across four queues.  Spreading the
> I/O across four queues instead of one just means that the device will
> process ¼ the rate of I/O from each of the four queues compared to if it
> was submitted on a single queue.
>
>
>
> For BlobFS, spreading the I/O across multiple NVMe queues would not
> normally help with latency.  There are NVMe features such as Weighted Round
> Robin (WRR), which provide different priorities to different queues.  With
> WRR, multiple NVMe queues could be used to separate high priority I/O (i.e.
> WAL writes) from lower priority I/O (i.e. background compaction I/O).  Most
> NVMe devices today do not support WRR however and even then it’s still
> questionable whether WRR alone would be sufficient or if additional
> software queuing would be required.
>

Is WRR a feature of device or a driver software feature?

if it's a device feature: Currently our research lab has two 300GB P3700
SSD. Will they support WRR?

If it's a software feature: Does SPDK block driver now support WRR? I am
guessing the BlobFS layer does not support WRR in that their is only one
dedicated core/thread/qpair doing the I/O. Is my understanding right?


>
>
> Thanks,
>
> -Jim
>
>
>
>
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Fenggang Wu <
> fenggang(a)cs.umn.edu>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, January 31, 2018 at 10:49 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Cc: *"wuxx0835(a)umn.edu" <wuxx0835(a)umn.edu>
> *Subject: *[SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O
> Threads
>
>
>
> Hi All,
>
>
>
> I read from the SPDK doc "NVMe Driver Design -- Scaling Performance" (here
> <http://www.spdk.io/doc/nvme.html#nvme_design>), which saids:
>
>
>
> " For example, if a device claims to be capable of 450,000 I/O per second
> at queue depth 128, in practice it does not matter if the driver is using 4
> queue pairs each with queue depth 32, or a single queue pair with queue
> depth 128."
>
>
>
> Does this consider the queuing latency? I am guessing the latency in the
> two cases will be different ( in qp/qd = 4/32 and in qp/qd = 1/128). In the
> 4 threads case, the latency will be 1/4 of the 1 thread case. Do I get it
> right?
>
>
>
> If so, then I got confused as the document also says:
>
>
>
> "In order to take full advantage of this scaling, applications should
> consider organizing their internal data structures such that data is
> assigned exclusively to a single thread."
>
>
>
> Please correct me if I get it wrong. I understand that if the dedicate I/O
> thread has the total ownership of the I/O data structures, there is no lock
> contention to slow down the I/O. I believe that BlobFS is also designed in
> this philosophy in that only one thread is doing I/O.
>
>
>
> But considering the RocksDB case, if the shared data structure has already
> been largely taken care of by the RocksDB logic via locking (which is
> inevitable anyway), the I/O requests each RocksDB thread sends to the
> BlobFS could also has its own queue pair to do I/O. More I/O threads means
> shorter queue depth and smaller queuing delay.
>
>
>
> Even if there is some FS metadata operations that may require some
> locking, but I would guest such metadata operation takes only a small
> portion.
>
>
>
> Therefore, is it a viable idea to have more I/O threads in the BlobFS to
> serve the multi-threaded RocksDB for a smaller delay? What will be the
> pitfalls, or challenges?
>
>
>
> Any thoughts/comments are appreciated. Thank you very much!
>
>
>
> Best!
>
> -Fenggang
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11272 bytes --]

Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 3174 bytes --]

Hi Fengang,

See below.

Regards,

-Jim


From: Fenggang Wu <wuxx0835(a)umn.edu>
Date: Wednesday, January 31, 2018 at 5:24 PM
To: James Harris <james.r.harris(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Performance Scaling in BlobFS/RocksDB by Multiple I/O Threads

Hi Jim,

I also have some quick followup question about WRR inline below.

Thank you very much!
-Fenggang

Fenggang WU(吴凤刚)
Ph.D. Student
Department of Computer Science and Engineering<http://www.cs.umn.edu/>
College of Science and Engineering<http://cse.umn.edu/>
University of Minnesota, Twin Cities<http://www.umn.edu>
Email: wuxx0835(a)umn.edu<mailto:wuxx0835(a)umn.edu>
Homepage: http://www.cs.umn.edu/~fenggang

On Wed, Jan 31, 2018 at 12:09 PM, Harris, James R <james.r.harris(a)intel.com<mailto:james.r.harris(a)intel.com>> wrote:
Hi Fenggang,

The max IOPs number is per-device – not per-queue.  The observed latency for each I/O - from submission to completion - will be the same whether the 128 I/O are submitted on one queue or across four queues.  Spreading the I/O across four queues instead of one just means that the device will process ¼ the rate of I/O from each of the four queues compared to if it was submitted on a single queue.

For BlobFS, spreading the I/O across multiple NVMe queues would not normally help with latency.  There are NVMe features such as Weighted Round Robin (WRR), which provide different priorities to different queues.  With WRR, multiple NVMe queues could be used to separate high priority I/O (i.e. WAL writes) from lower priority I/O (i.e. background compaction I/O).  Most NVMe devices today do not support WRR however and even then it’s still questionable whether WRR alone would be sufficient or if additional software queuing would be required.

Is WRR a feature of device or a driver software feature?

It is both.  WRR must first be supported by the device.  The AMS field in the CAP (Controller Capabilities) register specifies if the controller supports WRR.  I would recommend reading the section on Command Arbitration in the NVMe specification for additional details.

But the driver also requires changes to support WRR.  For example, spdk_nvme_ctrlr_alloc_io_qpair() takes an spdk_nvme_io_qpair_opts structure where the user can specify the priority for the allocated queue.

if it's a device feature: Currently our research lab has two 300GB P3700 SSD. Will they support WRR?

I do not believe the P3700 supports WRR – but you can check using the SPDK identify tool – examples/nvme/identify/identify.  Look for the section “Arbitration Mechanisms Supported”.

If it's a software feature: Does SPDK block driver now support WRR? I am guessing the BlobFS layer does not support WRR in that their is only one dedicated core/thread/qpair doing the I/O. Is my understanding right?

The SPDK bdev layer does support I/O prioritization currently.  If it did, BlobFS would still likely submit I/O from a single thread, but that thread would allocate multiple qpairs – one for each level of prioritization used.

Regards,

-Jim

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11222 bytes --]