End to end data integrity checking in Ceph?

End to end data integrity checking in Ceph?

[Craig Dunwoody]

Hi Sage,

-   Some of my customers are particularly paranoid about data integrity.
    I believe that there are some parallel/distributed storage systems
    (Panasas? Others?)  that claim to provide options for some kind of
    data integrity checking within the fault domain of the client, as a
    final step before delivering read-data to the application, and
    perhaps also as a first step after receiving write-data from the
    application.

    The hope would be to catch any data corruption that could possibly
    take place between clients and servers, even if relatively unlikely
    (e.g.  bits getting flipped in network transport).

-   Does the current Ceph implementation already do some variation of
    this?  If not, how difficult do you think it might be to add as a
    future optional feature?  Could it be added without breaking
    compatibility of wire-protocol and on-disk format?

Thanks in advance for any comments/suggestions on this.

Craig Dunwoody
GraphStream Incorporated

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

Re: End to end data integrity checking in Ceph?

[Sage Weil]

Hi Craig,

On Sun, 21 Mar 2010, Craig Dunwoody wrote:
> -   Some of my customers are particularly paranoid about data integrity.
>     I believe that there are some parallel/distributed storage systems
>     (Panasas? Others?)  that claim to provide options for some kind of
>     data integrity checking within the fault domain of the client, as a
>     final step before delivering read-data to the application, and
>     perhaps also as a first step after receiving write-data from the
>     application.
> 
>     The hope would be to catch any data corruption that could possibly
>     take place between clients and servers, even if relatively unlikely
>     (e.g.  bits getting flipped in network transport).
> 
> -   Does the current Ceph implementation already do some variation of
>     this?  If not, how difficult do you think it might be to add as a
>     future optional feature?  Could it be added without breaking
>     compatibility of wire-protocol and on-disk format?

The ceph transport layer does a crc32c over all data that passes over the 
wire to catch bit flips from the network (TCP's checksumming isn't very 
strong).  This isn't truly end-to-end protection, though, as bit flips on 
the client after the applicate write(2) but before writeback starts, or on 
the server after receiving the message won't be detected.

Btrfs does do it's own checksumming, so in theory if we match the function 
on the client we can do better.  There is also some end-to-end data 
integrity infrastructure in the kernel that IIRC Martin Peterson was 
working on.  Much that is in the block layer, though; the only parts that 
would be useful to ceph would relate to the userspace interface and page 
cache.  I'm not sure what the current state of that work is.

It would be nice to see end-to-end protection (complete with some sort of 
userspace api) in action on a local file system (probably btrfs, which 
actually stores checksums) as a model before trying to build it into a 
more complicated distributed file system...
 
sage

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

Re: End to end data integrity checking in Ceph?

[Craig Dunwoody]

Hello Sage,

sage writes:
>The ceph transport layer does a crc32c over all data that passes over the 
>wire to catch bit flips from the network (TCP's checksumming isn't very 
>strong).  This isn't truly end-to-end protection, though, as bit flips on 
>the client after the applicate write(2) but before writeback starts, or on 
>the server after receiving the message won't be detected.
>
>Btrfs does do it's own checksumming, so in theory if we match the function 
>on the client we can do better.  There is also some end-to-end data 
>integrity infrastructure in the kernel that IIRC Martin Peterson was 
>working on.  Much that is in the block layer, though; the only parts that 
>would be useful to ceph would relate to the userspace interface and page 
>cache.  I'm not sure what the current state of that work is.
>
>It would be nice to see end-to-end protection (complete with some sort of 
>userspace api) in action on a local file system (probably btrfs, which 
>actually stores checksums) as a model before trying to build it into a 
>more complicated distributed file system...

Thanks very much for this info.  Good to know that Ceph currently does
checking at network-transport level beyond what TCP does, and makes sense
to me that a local-FS implementation of end to end protection facilities
could be a next-step preceding distributed-FS implementation.

Craig Dunwoody
GraphStream Incorporated

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev