Read ahead affect Ceph read performance much

[Li Wang <liwang@ubuntukylin.com>]

We performed Iozone read test on a 32-node HPC server. Regarding the 
hardware of each node, the CPU is very powerful, so does the network, 
with a bandwidth > 1.5 GB/s. 64GB memory, the IO is relatively slow, the 
throughput measured by ‘dd’ locally is around 70MB/s. We configured a 
Ceph cluster with 24 OSDs on 24 nodes, one mds, one to four clients, one 
client per node. The performance is as follows,

     Iozone sequential read throughput (MB/s)
Number of clients     1          2         4
Default resize    180.0954   324.4836   591.5851
Resize: 256MB     645.3347   1022.998	1267.631

The complete iozone parameter for one client is,
iozone -t 1 -+m /tmp/iozone.nodelist.50305030 -s 64G -r 4M -i 0 -+n -w 
-c -e -b /tmp/iozone.nodelist.50305030.output, on each client node, only 
one thread is started.

for two clients, it is,
iozone -t 2 -+m /tmp/iozone.nodelist.50305030 -s 32G -r 4M -i 0 -+n -w 
-c -e -b /tmp/iozone.nodelist.50305030.output

As the data shown, a larger read ahead window could result in >300% speedup!

Besides, Since the backend of Ceph is not the traditional hard disk, it 
is beneficial to capture the stride read prefetching. To prove this, we 
tested the stride read with the following program, as we know, the 
generic read ahead algorithm of Linux kernel will not capture 
stride-read prefetch, so we use fadvise() to manually force pretching.
the record size is 4MB. The result is even more surprising,

             Stride read throughput (MB/s)
Number of records prefetched  0      1      4      16      64      128
Throughput                  42.82  100.74 217.41  497.73  854.48  950.18

As the data shown, with a read ahead size of 128*4MB, the speedup over
without read ahead could be up to 950/42 > 2000%!

The core logic of the test program is below,

stride = 17
recordsize = 4MB
for (;;) {
   for (i = 0; i < count; ++i) {
     long long start = pos + (i + 1) * stride * recordsize;
     printf("PRE READ %lld %lld\n", start, start + block);
     posix_fadvise(fd, start, block, POSIX_FADV_WILLNEED);
   }
   len = read(fd, buf, block);
   total += len;
   printf("READ %lld %lld\n", pos, (pos + len));
   pos += len;
   lseek(fd, (stride - 1) * block, SEEK_CUR);
   pos += (stride - 1) * block;
}

Given the above results and some more, We plan to submit a blue print to 
discuss the prefetching optimization of Ceph.

Cheers,
Li Wang




--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html