Re: [SPDK] A issue about maximums of write latency when we access the same block consecutively.

[-- Attachment #1: Type: text/plain, Size: 4557 bytes --]

Hi, Paul and Ben


We also considerate GC influences the write latencies, but have no ideas about how to demonstrate it by experiments.
Would you like to give us some advices about experiments design?
We also believe that it exists other factors which integrate to cause the abnormal phenomenon. 


Answers:
(1) "access" = write. We experiment read and write operations  respectively, 
      but only find the strange phenomenon in the writing experiments.
      The comparison of experiments can be seen in accessories.
(2) We use a NAND based SSD, Intel P3608.
(3) The result presented in accessories is produced with no delay. 
      We try to set "sleep(1)" between the two operations, but it seems does not work.
(4) We did not update the firmware...
     The device is shared by many fellows, we are afraid that the update is irreversible and affects other works.
     If it could be solved by firmware update, we also want to comprehend the reason?   


Thanks a lot.


At 2017-08-02 07:49:13, "Walker, Benjamin" <benjamin.walker(a)intel.com> wrote:
Hi Jiajia,


I have a bunch of questions that will help me figure out what you are seeing. 


1) When you say "access", do you mean read or write? The behavior of these two operations is quite different.
2) Are you using a NAND based or 3D XPoint based SSD? These again work entirely differently.
3) When you access the same block repeatedly, what's the delay between each access? None?
4) For whatever SSD you are using, have you confirmed the firmware is up to date? This can make a big difference.


Thanks, 
Ben




-------- Original message --------
From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: 8/1/17 12:42 PM (GMT-07:00)
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block consecutively.



Sounds like Danielle’s theory still holds here… the impact of background tasks in SSD firmware is at least partially dependent on the location of the operation on the media , ie if your write hits an area that is being relocated as part of GC it’s going to suffer more and as GC is done in localized areas it stands to reason that random traffic will be less affected as you’re hitting as a smaller % of the IOs will be hitting LBA regions that GC is affecting.  I’m sure there are a host of other factors as well

 

-Paul

 

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ?
Sent: Tuesday, August 1, 2017 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block consecutively.

 

Hi, Danielle

 

We also did experiments to access random blocks of 500 times, 

the results seem like more stable, most of the latencies are 10-30 μs and the maximum is 284 μs.

We have tested the random method  many times and each experiment has similar results.

 

We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause different influence between the random and consecutive accessing?      

Thanks a lot.

 

At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com> wrote:



What you are observing is the characteristics of how the drives firmware handles writes. This can be from write amplification caused by the drives GC, and journaling + checkpointing. This behavior may vary greatly between vendors and even firmware versions.

 

    - Danielle Costantino

From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 储 <cjj25233(a)163.com>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To:spdk(a)lists.01.org
Subject: [SPDK] A issue about maximums of write latency when we access the same block consecutively.

 

Hi, all

Recently, we use a demo to obverse the latency.

We find that when we access the same block consecutively, the occurrence of maximum latencies will become more frequent. 

Additionally,  they can reach even 2-3 ms and present a periodical change. 

 

 Why?

(1) For the same block, the latency of the first accessing is about 10-12 μs while the second, third 

     and the forth accessing can reach 700-900 μs even 2-3 ms? 

     I want to know the reason why the operation difference between the first accessing and the others exists.

(2) Why the maximums of 2-3 ms have a periodical change?

 

  Best wishes,

  Jiajia Chu

 

 

 

 

 

 

 

 

 

 

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12934 bytes --]

[-- Attachment #3: result_read.jpg --]
[-- Type: image/jpeg, Size: 39459 bytes --]

[-- Attachment #4: result_write.jpg --]
[-- Type: image/jpeg, Size: 49841 bytes --]