[dm-crypt] Memory errors and and 3TB RAID5 partition

[dm-crypt] Memory errors and and 3TB RAID5 partition

[Federico Foschini]

Hi all,
I'm reading the FAQ on cryptsetup wiki because I'm plannig to set up a
software raid 5 (mdadm) with 3 1,5TB disks and I'm a bit scared about
memory errors.  I read that a single flipped bit can lead in a massive
file corruption and I've got 2GB of non ECC ram on my machine... what
is the risk of encounter a memory error writing big files (like 8-10
GiB)?

--
Federico Foschini.

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Arno Wagner]

Hi Frederico,

impossible to quantify, and you may also get errors in the
busses, controller and (very rare unless overclocked) CPU.

The thing about "massive file corruption" is a myth though:
Most flipped bit just lead to a flipped bit because they 
happen before encryption. Even after encryption, you lose
only one 512 byte sector per flipped bit.

What you do is, if this is critical, _underclock_ everything in
your machine by 10-20%, get ECC RAM, ensure good cooling and clean 
power. Before undercklocking, test the machine extensively at 100% 
speed or maybe even 110% or the like.

An alternative is to verify each write. Here you have to be careful
to empty all buffers and caches before doing it. That is what I
do for large files. 

Arno

On Wed, Jan 16, 2013 at 12:08:15PM +0100, Federico Foschini wrote:
> Hi all,
> I'm reading the FAQ on cryptsetup wiki because I'm plannig to set up a
> software raid 5 (mdadm) with 3 1,5TB disks and I'm a bit scared about
> memory errors.  I read that a single flipped bit can lead in a massive
> file corruption and I've got 2GB of non ECC ram on my machine... what
> is the risk of encounter a memory error writing big files (like 8-10
> GiB)?
> 
> --
> Federico Foschini.
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Ralf Ramsauer]

Hi,

i have a 4x3TiB Raid 6(because it's expandable) setup using a md-raid +
dm-crypt + LVM.
Up to now, i never had any problems with it.

A friend just told me, that SATA has an error rate of about 1 Bit each
3TiB of transmission,
so a bit is written written or read wrong(but i don't know if this is
correct).
Worst case, one error bit leads to a corrupt raid sector(Raid 5 is able
to detect the corrupt
sector, but Raid 5 won't be able to correct it, correct me if i'm
wrong). This corrupt sector
could lead to a corrupt dm-crypt sector, which leads to an corrupt LVM
sector, and so on.
Finially, your filesystem could get broken.

So think about using large partition with encryption.

I never had trouble with my setup, but now I would set it up in some
other way.

Here's a nice article about it:
http://subnetmask255x4.wordpress.com/2008/10/28/sata-unrecoverable-errors-and-how-that-impacts-raid/
--
Ralf

On 01/16/13 12:08, Federico Foschini wrote:
> Hi all,
> I'm reading the FAQ on cryptsetup wiki because I'm plannig to set up a
> software raid 5 (mdadm) with 3 1,5TB disks and I'm a bit scared about
> memory errors.  I read that a single flipped bit can lead in a massive
> file corruption and I've got 2GB of non ECC ram on my machine... what
> is the risk of encounter a memory error writing big files (like 8-10
> GiB)?
>
> --
> Federico Foschini.
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Matthias Schniedermeyer]

On 16.01.2013 12:23, Arno Wagner wrote:
> Hi Frederico,
> 
> An alternative is to verify each write. Here you have to be careful
> to empty all buffers and caches before doing it. That is what I
> do for large files. 

My M.O. is to make MD5(*) sums of just about every file. So after i move 
files around, i just verify the MD5 sums. The amount of data moved is 
usually (a lot) bigger than available RAM, so i usually forgo clearing 
the cache, otherwise i do a umount/mount-cycle before verifying the MD5 
sums.

I'm doing this since a few years ago when i had trouble with a 
particular USB2<->Sata chip that had a rare tendency to corrupt a 
specific bit-pattern every now and then. Since then i had never any 
trouble with data-corruption.



*:
This case doesn't concern security, so MD5 is "good enough".

-- 

Matthias

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Federico Foschini]

Ok thank you all!

2013/1/16 Matthias Schniedermeyer <ms@citd.de>:
> On 16.01.2013 12:23, Arno Wagner wrote:
>> Hi Frederico,
>>
>> An alternative is to verify each write. Here you have to be careful
>> to empty all buffers and caches before doing it. That is what I
>> do for large files.
>
> My M.O. is to make MD5(*) sums of just about every file. So after i move
> files around, i just verify the MD5 sums. The amount of data moved is
> usually (a lot) bigger than available RAM, so i usually forgo clearing
> the cache, otherwise i do a umount/mount-cycle before verifying the MD5
> sums.
>
> I'm doing this since a few years ago when i had trouble with a
> particular USB2<->Sata chip that had a rare tendency to corrupt a
> specific bit-pattern every now and then. Since then i had never any
> trouble with data-corruption.
>
>
>
> *:
> This case doesn't concern security, so MD5 is "good enough".
>
> --
>
> Matthias
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt



-- 
Federico Foschini.

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Karol Babioch]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Hi,

Am 16.01.2013 13:00, schrieb Matthias Schniedermeyer:
> My M.O. is to make MD5(*) sums of just about every file.

What scripts and/or software do you use? I'm planning to do something
very similar, but wanted to make use of a file integrity checker. I
haven't looked too extensively, but the ones I came up with seemed to be
overkill.

I'm looking for something very basic, just enough to detect file
corruption and notify me by email or something like that.

Best regards,
Karol Babioch


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [dm-crypt] Memory errors and and 3TB RAID5 partition

[Matthias Schniedermeyer]

On 16.01.2013 17:30, Karol Babioch wrote:
> Hi,
> 
> Am 16.01.2013 13:00, schrieb Matthias Schniedermeyer:
> > My M.O. is to make MD5(*) sums of just about every file.
> 
> What scripts and/or software do you use? I'm planning to do something
> very similar, but wanted to make use of a file integrity checker. I
> haven't looked too extensively, but the ones I came up with seemed to be
> overkill.
> 
> I'm looking for something very basic, just enough to detect file
> corruption and notify me by email or something like that.

The software is a perl-script that i developed myself. Md5sum wasn't 
"intelligent" enough for me(*) and i didn't really look if there was any 
existing software. So i began with a small script around "Digest::MD5" 
and it grew from there. The script (i call it "md5tool.pl") is mostly 
generic, altough there are a few things specific for my environment 
hardcoded into it.




*:
For e.g. the script also records mtime, size and filetype with x/y-sizes 
for a few image-types.

With the mtime/size information the script can check if a file has been 
changed and update the stored MD5-sum.


-- 

Matthias