Re: CRC32 of messages

[Daniel Swarbrick <daniel.swarbrick@profitbricks.com>]

On 29/06/15 12:51, Gregory Farnum wrote:
> 
> Yes, we have our own CRC32 checksum because loooong ago (before I
> started!) Sage saw a lot of network corruption that wasn't being
> caught by the TCP checksums so he added some to the Ceph message
> stream. I can't tell you with any authority whatsoever how common that
> problem is, but I don't think we're turning them off by default in
> upstream. :)

If the CRC32 implementation in Ceph is that dated (particularly the
software implementations that will be used on AMD hardware), would it be
worth checking out some of the updated implementations, such as the
slice-by-16 or chunked methods?

I found this link http://create.stephan-brumme.com/crc32/ and tried
running the benchmark on an AMD Opteron 6386 SE system, with the
following results:

bitwise          : CRC=221F390F, 47.525s, 21.546 MB/s
half-byte        : CRC=221F390F, 11.828s, 86.576 MB/s
  1 byte  at once: CRC=221F390F, 6.347s, 161.332 MB/s
  4 bytes at once: CRC=221F390F, 2.875s, 356.178 MB/s
  8 bytes at once: CRC=221F390F, 2.004s, 510.932 MB/s
4x8 bytes at once: CRC=221F390F, 1.929s, 530.811 MB/s
 16 bytes at once: CRC=221F390F, 1.892s, 541.179 MB/s
 16 bytes at once: CRC=221F390F, 1.926s, 531.797 MB/s (including
prefetching)
    chunked      : CRC=221F390F, 1.919s, 533.656 MB/s

AFAIK, Ceph uses the slice-by-8 method if no hardware crc32 is found.