Re: [SPDK] A issue about maximums of write latency when we access the same block consecutively.

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

Hi, Danielle


We also did experiments to access random blocks of 500 times, 
the results seem like more stable, most of the latencies are 10-30 μs and the maximum is 284 μs.
We have tested the random method  many times and each experiment has similar results.


We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause different influence between the random and consecutive accessing?      
Thanks a lot.


At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com> wrote:


What you are observing is the characteristics of how the drives firmware handles writes. This can be from write amplification caused by the drives GC, and journaling + checkpointing. This behavior may vary greatly between vendors and even firmware versions.




    - Danielle Costantino
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 储 <cjj25233(a)163.com>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To:spdk(a)lists.01.org
Subject: [SPDK] A issue about maximums of write latency when we access the same block consecutively.
 
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum latencies will become more frequent. 
Additionally,  they can reach even 2-3 ms and present a periodical change. 


 Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the second, third 
     and the forth accessing can reach 700-900 μs even 2-3 ms? 
     I want to know the reason why the operation difference between the first accessing and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?


  Best wishes,
  Jiajia Chu




 





 





 





 

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3764 bytes --]

[-- Attachment #3: random_test_result.obj --]
[-- Type: application/octet-stream, Size: 4088 bytes --]

EAL: Detected 40 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
Initializing NVMe Controllers
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:06:00.0
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL:   probe driver: 8086:953 spdk_nvme
Attaching to 0000:07:00.0
Attached to 0000:06:00.0
Using controller INTEL SSDPECME032T4  (CVF8547400EL3P2CGN-1) with 1 namespaces.
  Namespace ID: 1 size: 1600GB
Attached to 0000:07:00.0
Using controller INTEL SSDPECME032T4  (CVF8547400EL3P2CGN-2) with 1 namespaces.
  Namespace ID: 1 size: 1600GB
Initialization complete.
37465 
284410 
14842 
10994 
14047 
11934 
16770 
11722 
9825 
10999 
15324 
10751 
122388 
9436 
9975 
8737 
10724 
8142 
8833 
8409 
8500 
14361 
8300 
12171 
9974 
8205 
10180 
9524 
10843 
27500 
10581 
8944 
9647 
8723 
9564 
11018 
13335 
9749 
9201 
9762 
7951 
7731 
10858 
7736 
12477 
10311 
8308 
7872 
10142 
15584 
6973 
10968 
8427 
11045 
9880 
7662 
11556 
10341 
8460 
9600 
11637 
87031 
8474 
8484 
7871 
7861 
12809 
9972 
8451 
10778 
12617 
9816 
9080 
11807 
8720 
8604 
9028 
12504 
7437 
8297 
11321 
9399 
8240 
7845 
7898 
8360 
10292 
9909 
8841 
11244 
10304 
9999 
7792 
7823 
27841 
10971 
11733 
10412 
9495 
8022 
10567 
10001 
8056 
12517 
7928 
10490 
12786 
9678 
9329 
22426 
7858 
7196 
6993 
8775 
12447 
11689 
10390 
11564 
8353 
17171 
8745 
12312 
10357 
8037 
7886 
9877 
82512 
11867 
8794 
17011 
10318 
7851 
12245 
7637 
10373 
8727 
10798 
10995 
7651 
11034 
7195 
11632 
8167 
9581 
7466 
22666 
11001 
8414 
7743 
7107 
7781 
10124 
7951 
7644 
11763 
8294 
10746 
8119 
7551 
9920 
12824 
6968 
8159 
9811 
9593 
14649 
9096 
8272 
9816 
7973 
10105 
9641 
9155 
9922 
12066 
9813 
9365 
8690 
11560 
10607 
9661 
11609 
11935 
9878 
7533 
10112 
10581 
7690 
13197 
10411 
7885 
73926 
7317 
8165 
11545 
8032 
8421 
11064 
8435 
8076 
12373 
9727 
13740 
7957 
11011 
11403 
9412 
8662 
8373 
10079 
8777 
10005 
7398 
7719 
9843 
12640 
9931 
8555 
7813 
10568 
10190 
7701 
7564 
7602 
26195 
10116 
10234 
8797 
10522 
8148 
8117 
8759 
11886 
9961 
8041 
10463 
13065 
10083 
10230 
8673 
10116 
11402 
11199 
15409 
8449 
8047 
10312 
9916 
12358 
7591 
7366 
7653 
8218 
8074 
10085 
7403 
7706 
12630 
18491 
9676 
9564 
8242 
7834 
9177 
10645 
12606 
9639 
7787 
10239 
10267 
7400 
12004 
8175 
9513 
9013 
8845 
7992 
13122 
7964 
7684 
10319 
7004 
7756 
7260 
16706 
10391 
12192 
8763 
7752 
7684 
7918 
10254 
7327 
11603 
8735 
11739 
10482 
7083 
10022 
7793 
7794 
10881 
12821 
8193 
10080 
11190 
10315 
10204 
8396 
17804 
11581 
9329 
10348 
15045 
9541 
9114 
12842 
12774 
8490 
9255 
8045 
81070 
7799 
18063 
11954 
10182 
8978 
9124 
10224 
8514 
7532 
10644 
9380 
8972 
7872 
10394 
7593 
9953 
9088 
7462 
10886 
7496 
8102 
21891 
8519 
8248 
10914 
15002 
11874 
9402 
7529 
7597 
10527 
10332 
8690 
10023 
19136 
7871 
9539 
7932 
7182 
12823 
7758 
11296 
8105 
7998 
8015 
9013 
10335 
11291 
8134 
10124 
8199 
9635 
9964 
10788 
8334 
10418 
13839 
7902 
10168 
7813 
10188 
10438 
13373 
9450 
80209 
8399 
16338 
8042 
7436 
7329 
10206 
10336 
13363 
10069 
7841 
10484 
9045 
10038 
8173 
7482 
9879 
9958 
9566 
10209 
9314 
10501 
7916 
7519 
9717 
7655 
7964 
9258 
8484 
13017 
7506 
7771 
20716 
28968 
10194 
10994 
10658 
8948 
10172 
10194 
9432 
8980 
7571 
9739 
12462 
7550 
10758 
11853 
10003 
9442 
7843 
10191 
7314 
9754 
7653 
8140 
12068 
11226 
8156 
9609 
9792 
9290 
10114 
10271 
7434 
9073 
10721 
9062 
10935 
9610 
13447 
8210 
10063 
7802 
7478 
10159 
10442 
8749 
8155 
6806 
9702 
7801 
8292 
7263 
10026 
7849 
9995 
8245 
9230 
7501 
7786 
7942 
13863 
11902 
10136 
9305 
9195 
9126 
27205 
10060 
8506 
9857 
8100 
7033 
7937 
10706 
7549 
8001 
8176 
10978 
7922 
11726 
9885 
8686