Data Integrity Check on EXT Family of Filesystems

Data Integrity Check on EXT Family of Filesystems

[Andrew Martin]

Hello,

I am considering writing a tool to perform data integrity checking on filesystems
which do not support it internally (e.g. ext4). When storing long-term backups, 
I would like to be able to detect bit rot or other corruption to ensure that I 
have intact backups. The method I am considering is to recreate the directory 
structure of the backup directory in a "shadow" directory tree, and then hash 
each of the files in the backup directory and store the hash in the same filename
in the shadow directory. Then, months later, I can traverse the backup directory,
taking a hash of each file again and comparing it with the hash stored in the 
shadow directory tree. If the hashes match, then the file's integrity has been
verified (or at least has not degraded since the shadow directory was created).

Does this seem like a reasonable approach for checking data integrity, or is there
an existing tool or different method which would be better?

Thanks,

Andrew Martin

Re: Data Integrity Check on EXT Family of Filesystems

[Mike Fleetwood]

On 23 September 2013 22:08, Andrew Martin <amartin@xes-inc.com> wrote:
>
> Hello,
>
> I am considering writing a tool to perform data integrity checking on filesystems
> which do not support it internally (e.g. ext4). When storing long-term backups,
> I would like to be able to detect bit rot or other corruption to ensure that I
> have intact backups. The method I am considering is to recreate the directory
> structure of the backup directory in a "shadow" directory tree, and then hash
> each of the files in the backup directory and store the hash in the same filename
> in the shadow directory. Then, months later, I can traverse the backup directory,
> taking a hash of each file again and comparing it with the hash stored in the
> shadow directory tree. If the hashes match, then the file's integrity has been
> verified (or at least has not degraded since the shadow directory was created).
>
> Does this seem like a reasonable approach for checking data integrity, or is there
> an existing tool or different method which would be better?
>
> Thanks,
>
> Andrew Martin

Here's a couple of integrity checking tools to consider:
tripwire - http://sourceforge.net/projects/tripwire/
aide - http://aide.sourceforge.net/

Don't use them, just providing options.

Thanks,
Mike



On 23 September 2013 22:08, Andrew Martin <amartin@xes-inc.com> wrote:
> Hello,
>
> I am considering writing a tool to perform data integrity checking on filesystems
> which do not support it internally (e.g. ext4). When storing long-term backups,
> I would like to be able to detect bit rot or other corruption to ensure that I
> have intact backups. The method I am considering is to recreate the directory
> structure of the backup directory in a "shadow" directory tree, and then hash
> each of the files in the backup directory and store the hash in the same filename
> in the shadow directory. Then, months later, I can traverse the backup directory,
> taking a hash of each file again and comparing it with the hash stored in the
> shadow directory tree. If the hashes match, then the file's integrity has been
> verified (or at least has not degraded since the shadow directory was created).
>
> Does this seem like a reasonable approach for checking data integrity, or is there
> an existing tool or different method which would be better?
>
> Thanks,
>
> Andrew Martin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Data Integrity Check on EXT Family of Filesystems

[Theodore Ts'o]

On Mon, Sep 23, 2013 at 11:00:09PM +0100, Mike Fleetwood wrote:
> 
> Here's a couple of integrity checking tools to consider:
> tripwire - http://sourceforge.net/projects/tripwire/
> aide - http://aide.sourceforge.net/

I use cfv myself to create checksum files.  Note that using CRC
checksum is plenty if you are just worried about random corruption
caused by hardware errors (i.e., memory bitflips, hard drive hiccups).

Using a cryptographic checksum ala tripwire is useful if you are
worried about malicious attackers trying to modify files without your
noticing.  Cryptographic checksums do take more CPU time, though.

Cheers,

					- Ted

Re: Data Integrity Check on EXT Family of Filesystems

[Zach Brown]

On Tue, Sep 24, 2013 at 10:57:24AM -0400, Theodore Ts'o wrote:
> On Mon, Sep 23, 2013 at 11:00:09PM +0100, Mike Fleetwood wrote:
> > 
> > Here's a couple of integrity checking tools to consider:
> > tripwire - http://sourceforge.net/projects/tripwire/
> > aide - http://aide.sourceforge.net/
> 
> I use cfv myself to create checksum files.  Note that using CRC
> checksum is plenty if you are just worried about random corruption
> caused by hardware errors (i.e., memory bitflips, hard drive hiccups).

And if one truly wanted hashes instead of checksums, it might not take
more than some light scripting around find and shasum -c.  KISS, and all
that.

- z

Re: Data Integrity Check on EXT Family of Filesystems

[Andrew Martin]

----- Original Message -----
> From: "Theodore Ts'o" <tytso@mit.edu>
> To: "Mike Fleetwood" <mike.fleetwood@googlemail.com>
> Cc: "Andrew Martin" <amartin@xes-inc.com>, linux-fsdevel@vger.kernel.org
> Sent: Tuesday, September 24, 2013 9:57:24 AM
> Subject: Re: Data Integrity Check on EXT Family of Filesystems
> 
> On Mon, Sep 23, 2013 at 11:00:09PM +0100, Mike Fleetwood wrote:
> > 
> > Here's a couple of integrity checking tools to consider:
> > tripwire - http://sourceforge.net/projects/tripwire/
> > aide - http://aide.sourceforge.net/
> 
> I use cfv myself to create checksum files.  Note that using CRC
> checksum is plenty if you are just worried about random corruption
> caused by hardware errors (i.e., memory bitflips, hard drive
> hiccups).
> 
> Using a cryptographic checksum ala tripwire is useful if you are
> worried about malicious attackers trying to modify files without your
> noticing.  Cryptographic checksums do take more CPU time, though.
> 
> Cheers,
> 
> 					- Ted
> 
Ted,

Thanks for the suggestion about cfv - it looks like it should work well 
for efficiently checking large trees and detecting random corruption. 
I like that the -r option stores the checksums recursively in each 
directory, that way you can also choose to only check a particular subtree.

Mike, thanks for the other suggestions as well. I've used tripwire 
before but had not considered using it in this context due to its 
focus on security and intrusion detection. 

Andrew