Disk I/O error while rebuilding an md raid-5 array

Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

Hi,

I have some trouble with my md raid-5 array.  It has four 500GB drives
(sdb1-sde1).  sde started giving some SMART error (Pending Bad
Sectors).  So just to be safe, I decided to replace it.  I declared it
to be faulty and removed it from the array and added a new drive and
the rebuilding was automatic.  But before the rebuilding can finish, I
got an I/O error from sdb1 and it was declared faulty by md.

Now I have two faulty drives and things don't look good.  However, I
was able to add the sdb back to the array and md seemed not mind and
still reported "active sync".  At this point I shut down computer and
decided to clone sdb with clonezilla so that I can have a good sdb to
finish rebuilding sde.  Not sure if it will complete without I/O
errors.  It appears clonezilla is using dd and the speed is extremely
slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.

The only reason I'm not in a total panic mode is that I did a back up
before doing all this.  Now I'm keeping my finger crossed that my
backup drive won't die.  In retrospect, I should have just shut down
the computer and cloned sde instead of letting md to rebuild the
array.

Any suggestions on the best way to proceed at this point are highly
appreciated, especially on the scenario that I won't be able to clone
sdb.  Is there any way to avoid building a new array?

Regards,

DS

PS, if in the end I have to build a new array, I'll probably go with a
raid 6 instead.

Re: Disk I/O error while rebuilding an md raid-5 array

[Greg Freemyer]

>
> PS, if in the end I have to build a new array, I'll probably go with a
> raid 6 instead.

Agreed, someone recently posted that for a raid-5 composed of 1TB
drives the odds of a rebuild failure are 1 in 67 even if the remaining
drives are within spec.  (ie. the unrecoverable bit error rate is
slowing succumbing to the ever increasing size of drives.)

You have 500GB drives, but you have 3 left to rebuild from, so that's
1.5 TB your trying to read.  I'm not sure how the original calculation
was done, so your odds of failed rebuild were either 1 in 134 or about
1 in 42.  Either not very good for something that is supposed to
protect your data.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com

Re: Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

On Mon, Feb 8, 2010 at 4:04 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
>>
>> PS, if in the end I have to build a new array, I'll probably go with a
>> raid 6 instead.
>
> Agreed, someone recently posted that for a raid-5 composed of 1TB
> drives the odds of a rebuild failure are 1 in 67 even if the remaining
> drives are within spec.  (ie. the unrecoverable bit error rate is
> slowing succumbing to the ever increasing size of drives.)

Has anyone done similar calculation for raid-6?

>
> Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Jon Hardcastle]

--- On Tue, 9/2/10, Dawning Sky <the.dawning.sky@gmail.com> wrote:

> From: Dawning Sky <the.dawning.sky@gmail.com>
> Subject: Re: Disk I/O error while rebuilding an md raid-5 array
> To: "Greg Freemyer" <greg.freemyer@gmail.com>
> Cc: linux-raid@vger.kernel.org
> Date: Tuesday, 9 February, 2010, 0:45
> On Mon, Feb 8, 2010 at 4:04 PM, Greg
> Freemyer <greg.freemyer@gmail.com>
> wrote:
> >>
> >> PS, if in the end I have to build a new array,
> I'll probably go with a
> >> raid 6 instead.
> >
> > Agreed, someone recently posted that for a raid-5
> composed of 1TB
> > drives the odds of a rebuild failure are 1 in 67 even
> if the remaining
> > drives are within spec.  (ie. the unrecoverable bit
> error rate is
> > slowing succumbing to the ever increasing size of
> drives.)
> 
> Has anyone done similar calculation for raid-6?
> 
> >
> > Greg

I am not sure of the math, but i was under the impression the odds were calculated PER DRIVE, not combining the array size. I HAD a 6 drive array comprised of 4x500GB and 2x750 (with some wasted space) as I wanted to push the new drives into 1~+ TB drives to make use of this lost space I decided to go RAID6 so i now have a 7 drive array.. 4x500GB 2x750 1x1TB. Now i am happy it is ok (see lots of earlier posts about mismatches) I am gearing up to replace 2 of the 500GB's with 1TB drives, and in 6 months I will repeat.

Bottom line, my 'gut' says if your drives are going to stray over 1TB go to RAID6 also, I have staggered DAILY smart reads (i think each drive gets 1 day a week 'off' and the rest are either long or short but never all doing 1 type) i also do a weekly 'scrub' of my array (this is so amazingly important).

My only downside is due to FSB speed issues i dont get more than 30m/s when accessing all 7 drives simultaneously but that is enough for RAID.


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[John Robinson]

On 09/02/2010 00:04, Greg Freemyer wrote:
>> PS, if in the end I have to build a new array, I'll probably go with a
>> raid 6 instead.
> 
> Agreed, someone recently posted that for a raid-5 composed of 1TB
> drives the odds of a rebuild failure are 1 in 67 even if the remaining
> drives are within spec.  (ie. the unrecoverable bit error rate is
> slowing succumbing to the ever increasing size of drives.)

Actually the odds were 1 in 67 of an unrecoverable read error while 
reading 2TB of data, if the odds were 1 in 10^15 per bit read[1], which 
was the worst-case spec offered by Western Digital. Others disagreed 
with my analysis, and I may be wrong.

This was nothing to do with RAID, but my suggestion followed on that 
RAID-5 was now only useful for defending against unrecoverable errors, 
and not dead drives, and if you wanted to defend against dead drives as 
well you need RAID-6.

> You have 500GB drives, but you have 3 left to rebuild from, so that's
> 1.5 TB your trying to read.  I'm not sure how the original calculation
> was done, so your odds of failed rebuild were either 1 in 134 or about
> 1 in 42.  Either not very good for something that is supposed to
> protect your data.

Actually there are 4 to read from - the original sde is still available. 
This would be a situation where I think having the hot-rebuild facility 
recently discussed on this list would be ideal, as if you can't read the 
data from the drive you're hot-replacing, you then get a second chance 
to read it from the rest of the drives using the parity information, and 
the odds of an unrecoverable read error at the same LBA on two drives is 
smaller - but I can't remember that bit of the probability course I did 
years ago to work out exactly what it is.

Cheers,

John.

[1] If the probability of an error while reading 1 bit is p, then the 
probability of an error while reading n bits is 1-(1-p)^n. In this case 
p=1E-15, n=1.6E13 and you need a scientific calculator to do the sum.

Re: Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

Thanks to everyone for the good advices.  Currently I plan to build a
5x500GB raid-6 from new drives and restore the data from the backup.
I'm keeping the original set of 4 offline just in case I need to
recover anything from it.

I still have a couple of questions I hope someone can help me with.

1. Can I still add the original sde back to the raid-5?  While I was
doing rebuilding, I believe some files were modified by the OS and the
old sde would be out of sync.

2. Quite a few people talked about "scrubbing" the raid, can someone
point me to somewhere I can read more about it?  I'm new to the list.
I tried googling and reading the list wiki and had a hard time finding
a tutorial for setting this up.

Regards,

DS

Re: Disk I/O error while rebuilding an md raid-5 array

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1007 bytes --]

On Tue, 9 Feb 2010, Dawning Sky wrote:

> 2. Quite a few people talked about "scrubbing" the raid, can someone 
> point me to somewhere I can read more about it?  I'm new to the list. I 
> tried googling and reading the list wiki and had a hard time finding a 
> tutorial for setting this up.

My debian/ubuntu boxes do this every sunday night via:

swmike@ub:/etc/cron.d$ cat mdadm
#
# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
#
# Copyright © martin f. krafft <madduck@madduck.net>
# distributed under the terms of the Artistic Licence 2.0
#

# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ] && /usr/share/mdadm/checkarray --cron --all --quiet



-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
> It appears clonezilla is using dd and the speed is extremely
> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.

BTW, is it normal for dd to be this slow, or is there something wrong
with my setup?  These are quite modern drives and should be able to
handle sequential transfer of 10's of MB/s.  I have never used dd
before so I don't have any experience.

DS

Re: Disk I/O error while rebuilding an md raid-5 array

[Wil Reichert]

On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
> On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
>> It appears clonezilla is using dd and the speed is extremely
>> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>
> BTW, is it normal for dd to be this slow, or is there something wrong
> with my setup?  These are quite modern drives and should be able to
> handle sequential transfer of 10's of MB/s.  I have never used dd
> before so I don't have any experience.

dd should be pretty close to the max possible throughput of yer drive.
 For example, I dd'd a 1T WD black not too long ago & it was around
120MB/s on the outside & 70MB/s in the middle.

Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Steven Haigh]

On Mon, 8 Feb 2010 22:24:09 -0600, Wil Reichert <wil.reichert@gmail.com>
wrote:
> On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky <the.dawning.sky@gmail.com>
> wrote:
>> On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com>
>> wrote:
>>> It appears clonezilla is using dd and the speed is extremely
>>> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>>
>> BTW, is it normal for dd to be this slow, or is there something wrong
>> with my setup?  These are quite modern drives and should be able to
>> handle sequential transfer of 10's of MB/s.  I have never used dd
>> before so I don't have any experience.
> 
> dd should be pretty close to the max possible throughput of yer drive.
>  For example, I dd'd a 1T WD black not too long ago & it was around
> 120MB/s on the outside & 70MB/s in the middle.

I think this depends more on the block size that is used. 1MB block sizes
are much faster than 512 byte blocks...

-- 
Steven Haigh
 
Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Wil Reichert]

On Mon, Feb 8, 2010 at 10:26 PM, Steven Haigh <netwiz@crc.id.au> wrote:
> On Mon, 8 Feb 2010 22:24:09 -0600, Wil Reichert <wil.reichert@gmail.com>
> wrote:
>> On Mon, Feb 8, 2010 at 7:23 PM, Dawning Sky <the.dawning.sky@gmail.com>
>> wrote:
>>> On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com>
>>> wrote:
>>>> It appears clonezilla is using dd and the speed is extremely
>>>> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>>>
>>> BTW, is it normal for dd to be this slow, or is there something wrong
>>> with my setup?  These are quite modern drives and should be able to
>>> handle sequential transfer of 10's of MB/s.  I have never used dd
>>> before so I don't have any experience.
>>
>> dd should be pretty close to the max possible throughput of yer drive.
>>  For example, I dd'd a 1T WD black not too long ago & it was around
>> 120MB/s on the outside & 70MB/s in the middle.
>
> I think this depends more on the block size that is used. 1MB block sizes
> are much faster than 512 byte blocks...

Good point, I was using bs=1M.

Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
> Hi,
>
> Now I have two faulty drives and things don't look good.  However, I
> was able to add the sdb back to the array and md seemed not mind and
> still reported "active sync".  At this point I shut down computer and
> decided to clone sdb with clonezilla so that I can have a good sdb to
> finish rebuilding sde.  Not sure if it will complete without I/O
> errors.  It appears clonezilla is using dd and the speed is extremely
> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>

As expected, dd encountered the same UNC error.  Now I'm trying to
ddrescue the drive to see what happens.  My question is whether this
is worth doing.  Assuming ddrescue cannot read the bad sector either
and writes 0's to the new drive, will I be able to rebuild the raid-5,
from 2 good disks and this disk with a bad sector?  I can assume there
will be a bad file but will the array still function?

Or I'm better off just build a new array from scratch.

Any suggestions are appreciated.

Regards,

DS
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Stefan Hübner]

Hi!

I do RAID-recoveries at least once a month and get paid for it.  Rule of
thumb: if your have one drive dropped and another one with pending
sectors, your rebuild will fail - no need for calculations there.

ddrescue on a clean disk is about half as fast as dd with a blocksize
beyond 1M.  ddrescue on a disk with pending sectors is just no pita as
dd or sg_dd would be, because it adds the neccesary intelligence.

Do you have the original sde still around?  If yes, ddrescue both: sdb
and sde.  My experience says: there will only be a few KB lost.  Then
re-create your raid (it will only write the superblock new) with
"--assume-clean".  After that worked, you might make another (big)
backup first, then run fsck and see what happens.  If the lost bytes
have screwed the filesystem, you might want to re-create the raid with
another (personally I prefer xfs) fs and replay your backup into it.

A few commands to make the intentions cleaner:
ddrescue -dv -r5 /dev/oldsdb1 /dev/newsdb1 /root/sdblog
ddrescue -dv -r5 /dev/oldsde1 /dev/newsde1 /root/sdelog
... find out which drive is which raid-device -> mdadm -E /dev/sdX1
mdadm --create /dev/md0 --raid-devices=4 --level=5
--chunk=${your_chunk_size_in_kb} --assume-clean
${ordered_list_of_raid_devices}

Hope this helps,
Stefan Hübner


Am 09.02.2010 05:20, schrieb Dawning Sky:
> On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
>   
>> Hi,
>>
>> Now I have two faulty drives and things don't look good.  However, I
>> was able to add the sdb back to the array and md seemed not mind and
>> still reported "active sync".  At this point I shut down computer and
>> decided to clone sdb with clonezilla so that I can have a good sdb to
>> finish rebuilding sde.  Not sure if it will complete without I/O
>> errors.  It appears clonezilla is using dd and the speed is extremely
>> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>>
>>     
> As expected, dd encountered the same UNC error.  Now I'm trying to
> ddrescue the drive to see what happens.  My question is whether this
> is worth doing.  Assuming ddrescue cannot read the bad sector either
> and writes 0's to the new drive, will I be able to rebuild the raid-5,
> from 2 good disks and this disk with a bad sector?  I can assume there
> will be a bad file but will the array still function?
>
> Or I'm better off just build a new array from scratch.
>
> Any suggestions are appreciated.
>
> Regards,
>
> DS
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Dawning Sky]

Thanks for the good advice.  ddrescue on sdb returned an error of 4kB.
 I do still have my old sde.  But one thing I did, which was stupid,
was trying to rebuild the raid-5 when it was mounted.  So I don't know
the old sde is still consistent with the rest 3 disks, since some
files would have been modified between the times when I took the old
sde offline and when the rebuild failed.

So at this point, I guess I'll get 4 new drives and set up a brand new
raid-6 and try to restore my data from my backup in an external drive
and hope for the best.  I'll keep the 4 drives from my old raid-5 just
in case if I need to recover something from them.

I guess I learned my lesson.  I should have ddrescued all the disks I
want to replace, instead of using md's rebuild mechanism.

DS


On Mon, Feb 8, 2010 at 10:57 PM, Stefan Hübner
<stefan.huebner@stud.tu-ilmenau.de> wrote:
> Hi!
>
> I do RAID-recoveries at least once a month and get paid for it.  Rule of
> thumb: if your have one drive dropped and another one with pending
> sectors, your rebuild will fail - no need for calculations there.
>
> ddrescue on a clean disk is about half as fast as dd with a blocksize
> beyond 1M.  ddrescue on a disk with pending sectors is just no pita as
> dd or sg_dd would be, because it adds the neccesary intelligence.
>
> Do you have the original sde still around?  If yes, ddrescue both: sdb
> and sde.  My experience says: there will only be a few KB lost.  Then
> re-create your raid (it will only write the superblock new) with
> "--assume-clean".  After that worked, you might make another (big)
> backup first, then run fsck and see what happens.  If the lost bytes
> have screwed the filesystem, you might want to re-create the raid with
> another (personally I prefer xfs) fs and replay your backup into it.
>
> A few commands to make the intentions cleaner:
> ddrescue -dv -r5 /dev/oldsdb1 /dev/newsdb1 /root/sdblog
> ddrescue -dv -r5 /dev/oldsde1 /dev/newsde1 /root/sdelog
> ... find out which drive is which raid-device -> mdadm -E /dev/sdX1
> mdadm --create /dev/md0 --raid-devices=4 --level=5
> --chunk=${your_chunk_size_in_kb} --assume-clean
> ${ordered_list_of_raid_devices}
>
> Hope this helps,
> Stefan Hübner
>
>
> Am 09.02.2010 05:20, schrieb Dawning Sky:
>> On Mon, Feb 8, 2010 at 3:23 PM, Dawning Sky <the.dawning.sky@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Now I have two faulty drives and things don't look good.  However, I
>>> was able to add the sdb back to the array and md seemed not mind and
>>> still reported "active sync".  At this point I shut down computer and
>>> decided to clone sdb with clonezilla so that I can have a good sdb to
>>> finish rebuilding sde.  Not sure if it will complete without I/O
>>> errors.  It appears clonezilla is using dd and the speed is extremely
>>> slow (~5MB/sec) and it says it's gonna take 1 day to clone the 500GB.
>>>
>>>
>> As expected, dd encountered the same UNC error.  Now I'm trying to
>> ddrescue the drive to see what happens.  My question is whether this
>> is worth doing.  Assuming ddrescue cannot read the bad sector either
>> and writes 0's to the new drive, will I be able to rebuild the raid-5,
>> from 2 good disks and this disk with a bad sector?  I can assume there
>> will be a bad file but will the array still function?
>>
>> Or I'm better off just build a new array from scratch.
>>
>> Any suggestions are appreciated.
>>
>> Regards,
>>
>> DS
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Disk I/O error while rebuilding an md raid-5 array

[Mikael Abrahamsson]

On Mon, 8 Feb 2010, Dawning Sky wrote:

> Thanks for the good advice.  ddrescue on sdb returned an error of 4kB.
> I do still have my old sde.  But one thing I did, which was stupid,
> was trying to rebuild the raid-5 when it was mounted.  So I don't know
> the old sde is still consistent with the rest 3 disks, since some
> files would have been modified between the times when I took the old
> sde offline and when the rebuild failed.

Please see the threads I have initiated the past 2-3 weeks, I had the 
exact same problem.

What you should do is just put in your new cloned drive, make sure it runs 
fine with 3 drives, and then start the rebuild with the drive that you 
were resyncing to. I've done this numerous times, and my experience says 
you will only have lost those 4 kilobytes of data that ddrescue couldn't 
read.

I'm now happily at RAID6 after doing a re-shape from 6 drive raid5-> 8 
drive raid6, change of SATA cables and a new marvell 8channel 
controller... No more read timeouts anyway, and I should be better 
protected from single read errors like you are having.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Disk I/O error while rebuilding an md raid-5 array

[russ]

I had no idea the odds were that bad.  Time to switch to zfs...

Russ
------Original Message------
From: Greg Freemyer
Sender: linux-raid-owner@vger.kernel.org
To: Dawning Sky
Cc: linux-raid@vger.kernel.org
Subject: Re: Disk I/O error while rebuilding an md raid-5 array
Sent: Feb 8, 2010 7:04 PM

>
> PS, if in the end I have to build a new array, I'll probably go with a
> raid 6 instead.

Agreed, someone recently posted that for a raid-5 composed of 1TB
drives the odds of a rebuild failure are 1 in 67 even if the remaining
drives are within spec.  (ie. the unrecoverable bit error rate is
slowing succumbing to the ever increasing size of drives.)

You have 500GB drives, but you have 3 left to rebuild from, so that's
1.5 TB your trying to read.  I'm not sure how the original calculation
was done, so your odds of failed rebuild were either 1 in 134 or about
1 in 42.  Either not very good for something that is supposed to
protect your data.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Sent from my Verizon Wireless BlackBerry