RAID backup

RAID backup

[jbradford]

Might it not be a good idea to DD the raw contents of each disk to a tape drive, just incase you fubar the array?  It would be time consuming, but at least you could restore your data in the event that it gets corrupted.

John.

Re: RAID backup

[Roy Sigurd Karlsbakk]

On Thursday 03 October 2002 13:20, jbradford@dial.pipex.com wrote:
> Might it not be a good idea to DD the raw contents of each disk to a tape
> drive, just incase you fubar the array?  It would be time consuming, but at
> least you could restore your data in the event that it gets corrupted.

er

16 120GB disks?
-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: RAID backup

[jbradford]

> > Might it not be a good idea to DD the raw contents of each disk to a tape
> > drive, just incase you fubar the array?  It would be time consuming, but at
> > least you could restore your data in the event that it gets corrupted.
> 
> er
> 
> 16 120GB disks?

A fast, large tape backup :-)

Seriously, if this is crucial data, (E.G. solution to Fermat's last theorum), I was just pointing out that you can backup, even without mounting the disks.

John.

Re: RAID backup

[Kanoalani Withington]

I have to pipe in here and agree that the idea of using a disk array 
alone for backups is not a sound idea. Sure, backing up 2Tb to an old 
exabyte drive isn't going to work, if you really have that much data you 
need some more modern equipment.

Essentially I believe the idea of a redundant array sounds safer than it 
really is in practice, especially when dealing with very large arrays 
and with level 5 arrays. The reasons why this is so are manifold, 
suffice to say that a few years of actually using such devices shows 
that they have much more potential for catastrophic failure and latent 
failure (you don't know it's broken until you go to use it and find out 
it's broken) than a well designed tape archive or backup.

Not that disk to disk backups are a completely bad idea. In my 
experience a combination works best. For example, automatic backups to 
reserved disks or disk arrays on remote systems every night, but once a 
week tape snapshots of that data. It's a lot of tapes but over time it 
will prove to be worthwhile. If the data volume is too high, simple 
backup scripts that write every file only once (essentially an archive) 
to tape to make it more practical.

-Kanoa


Roy Sigurd Karlsbakk wrote:

>On Thursday 03 October 2002 13:20, jbradford@dial.pipex.com wrote:
>
>>Might it not be a good idea to DD the raw contents of each disk to a tape
>>drive, just incase you fubar the array?  It would be time consuming, but at
>>least you could restore your data in the event that it gets corrupted.
>>
>
>er
>
>16 120GB disks?
>

RE: RAID backup

[Effrem Norwood]

> I have to pipe in here and agree that the idea of using a disk array
> alone for backups is not a sound idea. Sure, backing up 2Tb to an old
> exabyte drive isn't going to work, if you really have that much data you
> need some more modern equipment.

I've worked in the storage industry for years and surprisingly more and more
organizations *are* moving to disk only backup strategies with a lot of good
reasons for doing so. Large companies are replacing 100% of their tape
infrastructure with multiple large yet inexpensive disk array's. Disk backup
infrastructure prices are now equal to or lower than tape backup
infrastructure prices on initial acquisition, and significantly lower if you
factor in ROI and TCO - not to mention huge backup/restore time savings and
potential disaster recovery possibilities.

> Essentially I believe the idea of a redundant array sounds safer than it
> really is in practice, especially when dealing with very large arrays
> and with level 5 arrays. The reasons why this is so are manifold,
> suffice to say that a few years of actually using such devices shows
> that they have much more potential for catastrophic failure and latent
> failure (you don't know it's broken until you go to use it and find out
> it's broken) than a well designed tape archive or backup.

With multiple inexpensive large disk arrays from companies like Network
Appliance (NearStor) and Exstor (T-2120) organizations are asynchronously
mirroring their data to geographically distant locations to prevent single
points of failure with their arrays. As you suggest, a single array can fail
and lose data. From a tape perspective, a tape in a backup set can fail as
well-potentially presenting the same catastrophic situation of not being
able to recover. Just as organizations have multiple sets of tapes on a
rotation schedule, organizations now often have multiple large near-line
storage arrays with a data rotation schedule.

> Not that disk to disk backups are a completely bad idea. In my
> experience a combination works best. For example, automatic backups to
> reserved disks or disk arrays on remote systems every night, but once a
> week tape snapshots of that data. It's a lot of tapes but over time it
> will prove to be worthwhile. If the data volume is too high, simple
> backup scripts that write every file only once (essentially an archive)
> to tape to make it more practical.

Classical Hierarchical Storage Management (HSM) as you describe here is
being undermined by the fact that these large IDE based disk arrays are
equally or less expensive than tapes. In addition, HSM software costs
something (even if you write it yourself) on top of the tape infrastructure.
One customer of ours was quoted 40K per TB of HSM *software* alone. Well
designed tape replacement disk systems can cost 25K per TB or less and have
many advantages over tape. Online data, faster data access, and disaster
recovery just to name a few. Further, backup software vendors like Veritas
and Legato now recognize disks as a viable tape replacement and are
incorporating that fact into their software.

If tape is absolutely required, the infrastructure required per TB of taped
data can be significantly smaller and the backup window can be massively
extended - from hours to weeks with a large near-line disk approach. The
tape/library cost component in this way is significantly reduced.

For purposes of full disclosure, I work at Exstor and we provide tape
replacement solutions as well as high performance NAS solutions for the
enterprise.

Regards,

Eff Norwood

Re: RAID backup

[Lars Marowsky-Bree]

On 2002-10-03T16:59:57,
   Effrem Norwood <enorwood@effrem.com> said:

> With multiple inexpensive large disk arrays from companies like Network
> Appliance (NearStor) and Exstor (T-2120) organizations are asynchronously
> mirroring their data to geographically distant locations to prevent single
> points of failure with their arrays.

Let's just point out that Linux can do that too with drbd.

(I wonder if that will stay a separate module or whether it'll become a EVMS
plugin ;-)


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Principal Squirrel
Research and Development, SuSE Linux AG
 
``Immortality is an adequate definition of high availability for me.''
	--- Gregory F. Pfister

Re: RAID backup

[Illtud Daniel]

Effrem Norwood wrote:

> In addition, HSM software costs
> something (even if you write it yourself) on top of the tape infrastructure.
> One customer of ours was quoted 40K per TB of HSM *software* alone.

I've got 25TB uncompressed of HSM here, and it's cost us (ex VAT)
roughly:

£10k for the first 1.6TB (on NT, DLT library)
£18k for the next  6.0TB (on NT, LTO library)
£45k for the next 18.0TB (on Solaris, LTO Library)

...for the software licencing alone. Plus about 10% pa. in support
costs.
You're looking at about 50-60% of the library cost for the HSM software
to manage it (tapes are another thing). Is HSM really that difficult?

It really is a racket, but it's not so much compared with the
cost of re-producing the data (mainly digitized collections).
I'd be happier about it if they were more reliable (libs and s/w).
Disk arrays, on the other hand, would cost us a fortune in
upgrading the cooling - we've had to do this once just because of
the 3-4 TB of online storage we've got, and adding huge exchangers
(and associated pipes) isn't something I want to do much of.

Oh, and having spent much of last night and this morning dealing
with multiple SCSI disk failures, and having seen about 5% of
ours fail in a year, I'm rapidly seeing the light on IDE.

-- 
Illtud Daniel                                 illtud.daniel@llgc.org.uk
Uwch Ddadansoddwr Systemau                       Senior Systems Analyst
Llyfrgell Genedlaethol Cymru                  National Library of Wales
Yn siarad drosof fy hun, nid LlGC   -  Speaking personally, not for NLW
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID backup

[jbradford]

> > In addition, HSM software costs
> > something (even if you write it yourself) on top of the tape infrastructure.
> > One customer of ours was quoted 40K per TB of HSM *software* alone.
> 
> I've got 25TB uncompressed of HSM here, and it's cost us (ex VAT)
> roughly:
> 
> £10k for the first 1.6TB (on NT, DLT library)
> £18k for the next  6.0TB (on NT, LTO library)
> £45k for the next 18.0TB (on Solaris, LTO Library)
> 
> ..for the software licencing alone. Plus about 10% pa. in support
> costs.
> You're looking at about 50-60% of the library cost for the HSM software
> to manage it (tapes are another thing). Is HSM really that difficult?
> 
> It really is a racket, but it's not so much compared with the
> cost of re-producing the data (mainly digitized collections).
> I'd be happier about it if they were more reliable (libs and s/w).
> Disk arrays, on the other hand, would cost us a fortune in
> upgrading the cooling - we've had to do this once just because of
> the 3-4 TB of online storage we've got, and adding huge exchangers
> (and associated pipes) isn't something I want to do much of.
> 
> Oh, and having spent much of last night and this morning dealing
> with multiple SCSI disk failures, and having seen about 5% of
> ours fail in a year, I'm rapidly seeing the light on IDE.

This is rapidly becoming off topic, especially for the kernel dev list, which is why I originally just posted the following info to the cc'ed people, but since it might be of interest, I'm posting it to the lists:

Sony GY-8240-DTF2 tape drive, picture of it, with some info here:

http://www.tomtec.co.jp/english/tape_hisped.html

this uses similar technology to that which is used in their Digital Betacam(tm) studio VCRs, and stores 200 Gigs uncompressed on one large tape, (60 on a small tape).

Data rate, 24 MB/s.

You've got to have a lot of data for that to be insufficient, and it would be my backup system of choice, if I needed the capcity, (I currently use punched Myler tape :-) ).

John.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID backup

[Alvin Oga]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=US-ASCII, Size: 987 bytes --]


hi ya

we can build an 8-drive ( 120GB at $200ea )  or ( 160GB at $300>? each )
1U box...about 0.960 - 1.28 TB each backup server ( 1U ) for under $2,500 in parts
+  cost of raid setup/testing is up to the user
	- am thinking the 1.6TB of storage for 10K lira(?) is too much

i prefer disks to backup data.. so that its always a semi-warm backup
( tapes have always been way tooo slow to find a file and to restore 

have fun
alvin
http://www.Linux-1U.net

On Fri, 4 Oct 2002, Illtud Daniel wrote:

> Effrem Norwood wrote:
> 
> > In addition, HSM software costs
> > something (even if you write it yourself) on top of the tape infrastructure.
> > One customer of ours was quoted 40K per TB of HSM *software* alone.
> 
> I've got 25TB uncompressed of HSM here, and it's cost us (ex VAT)
> roughly:
> 
> £10k for the first 1.6TB (on NT, DLT library)
> £18k for the next  6.0TB (on NT, LTO library)
> £45k for the next 18.0TB (on Solaris, LTO Library)
> 
> 

Re: RAID backup

[Alan Cox]

On Fri, 2002-10-04 at 12:20, Alvin Oga wrote:
> we can build an 8-drive ( 120GB at $200ea )  or ( 160GB at $300>? each )
> 1U box...about 0.960 - 1.28 TB each backup server ( 1U ) for under $2,500 in parts
> +  cost of raid setup/testing is up to the user
> 	- am thinking the 1.6TB of storage for 10K lira(?) is too much
> 
> i prefer disks to backup data.. so that its always a semi-warm backup
> ( tapes have always been way tooo slow to find a file and to restore 

The problem with disks is you still have to archive them somewhere, and
they are bulky. I also dont know what studies are available on the
degradation of stored disk media over time. 

Capacity is not a problem, 3ware do a 12 channel sata card, with maxtor
drives that comes in at 320x12 = 3.5Tb

Re: RAID backup

[Mr. James W. Laferriere]

	Hello Alan ,  I do not know about modern disk media .  But I had a
	Seagate st501 drive that sat shelved for 5 years & put it back
	into the micro-pdp11/23 & booted fine .  I ran that for ~ another
	year for a small project I was involved in .  I would hope that
	more modern media would have better shelf life than that even ?-)
		Twyl ,  JimL

On 4 Oct 2002, Alan Cox wrote:
> On Fri, 2002-10-04 at 12:20, Alvin Oga wrote:
> > we can build an 8-drive ( 120GB at $200ea )  or ( 160GB at $300>? each )
> > 1U box...about 0.960 - 1.28 TB each backup server ( 1U ) for under $2,500 in parts
> > +  cost of raid setup/testing is up to the user
> > 	- am thinking the 1.6TB of storage for 10K lira(?) is too much
> > i prefer disks to backup data.. so that its always a semi-warm backup
> > ( tapes have always been way tooo slow to find a file and to restore

> The problem with disks is you still have to archive them somewhere, and
> they are bulky. I also dont know what studies are available on the
> degradation of stored disk media over time.

> Capacity is not a problem, 3ware do a 12 channel sata card, with maxtor
> drives that comes in at 320x12 = 3.5Tb

       +------------------------------------------------------------------+
       | James   W.   Laferriere | System    Techniques | Give me VMS     |
       | Network        Engineer |     P.O. Box 854     |  Give me Linux  |
       | babydr@baby-dragons.com | Coudersport PA 16915 |   only  on  AXP |
       +------------------------------------------------------------------+

Re: RAID backup

[Dr. David Alan Gilbert]

* Alan Cox (alan@lxorguk.ukuu.org.uk) wrote:
> 
> The problem with disks is you still have to archive them somewhere, and
> they are bulky. I also dont know what studies are available on the
> degradation of stored disk media over time. 

Not sure about that; DLT tapes are pretty bulky themselves; I think the
difference between say a set of 4 DLT tapes and a single Maxtor 320 in
caddy would be minimal. As for stored media, I think Maxtor are quoting
1M hours MTTF - (I hate to think how you measure such a figure) - for
the 320G, and that is probably longer than I'd trust either the tape or
the drive to survive.

> Capacity is not a problem, 3ware do a 12 channel sata card, with maxtor
> drives that comes in at 320x12 = 3.5Tb

Well to me there are two questions:
  1) Price with caddy/drive - especially when you need to have
	multiple backup sets.

	2) Linux serial/ata working reliably with hot swapping.

If both those came out on the right side then I'd happily swap to discs.

Dave
 ---------------- Have a happy GNU millennium! ----------------------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM, SPARC and HP-PA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Re: RAID backup

[Russell King]

On Fri, Oct 04, 2002 at 02:24:19PM +0100, Dr. David Alan Gilbert wrote:
> Not sure about that; DLT tapes are pretty bulky themselves; I think the
> difference between say a set of 4 DLT tapes and a single Maxtor 320 in
> caddy would be minimal.

The 4 DLT tapes would take up twice the room.

> As for stored media, I think Maxtor are quoting
> 1M hours MTTF - (I hate to think how you measure such a figure) - for
> the 320G, and that is probably longer than I'd trust either the tape or
> the drive to survive.

However, drive in caddy or no caddy, the accidental drop test would
probably be more favourable to the DLT tape surviving over the drive.
Physical accidents do happen.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: RAID backup

[Dr. David Alan Gilbert]

* Russell King (rmk@arm.linux.org.uk) wrote:

> However, drive in caddy or no caddy, the accidental drop test would
> probably be more favourable to the DLT tape surviving over the drive.
> Physical accidents do happen.

While I guess a good caddy might put a lump of rubber in to help. But
there again I'm not sure if I have that much confidence in the DLT tapes
either; the instructions in the insert for HP DLT tapes tell you to
rattle them and listen to hear if you can hear anything loose
before putting them into your drive!

Anyway, from the side of data integrity the drop test doesn't worry me -
for critical data I have a lot more than one backup; and users perform
an important part of the backup test system by regularly deleting files
to be restored.

Dave

 ---------------- Have a happy GNU millennium! ----------------------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM, SPARC and HP-PA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/

Re: RAID backup

[Alvin Oga]

hi ya

mroe fun stuff

"drop tests" ... that person that tend to have "accidents" should be
kept outof the lab and away from hardware
	( ceo/cfo/managers dont wanna spend un-necessary $2K - $10k
	( because somebody dropped it ... or dropped a screw in a running
	( system

on the other hand... that clumbsy tech/engineer is purrfect ina
testing lab or qa lab where they do require drop tests for "submarine"
applications

have fun
alvin

- i never dropped a disks in 20 years... or mb or anything
  importnat/expensive

- dropped lots of screws, screwdrivers into "turned off systems" while
  working on it

- dropped tape drives too and sometimes the tape drives itself ejects the
  tapes too too fast
	- clumbsy me for not being able to catch the 'flying tape" as
	it ejects

- a months worth of daily tapes ( we keep 30 days worth of tape )
	takes up as much room as a 1U box.. 

	-  a 1U box can hold about 3 months worth of compressed data
	in our environment ... ( 3:1 compression on the average )


On Fri, 4 Oct 2002, Russell King wrote:

> On Fri, Oct 04, 2002 at 02:24:19PM +0100, Dr. David Alan Gilbert wrote:
> > Not sure about that; DLT tapes are pretty bulky themselves; I think the
> > difference between say a set of 4 DLT tapes and a single Maxtor 320 in
> > caddy would be minimal.
> 
> The 4 DLT tapes would take up twice the room.
> 
> > As for stored media, I think Maxtor are quoting
> > 1M hours MTTF - (I hate to think how you measure such a figure) - for
> > the 320G, and that is probably longer than I'd trust either the tape or
> > the drive to survive.
> 
> However, drive in caddy or no caddy, the accidental drop test would
> probably be more favourable to the DLT tape surviving over the drive.
> Physical accidents do happen.
> 
> -- 
> Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
>              http://www.arm.linux.org.uk/personal/aboutme.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: RAID backup

[Luca Berra]

Dr. David Alan Gilbert wrote:
> * Alan Cox (alan@lxorguk.ukuu.org.uk) wrote:
> 
>>The problem with disks is you still have to archive them somewhere, and
>>they are bulky. I also dont know what studies are available on the
>>degradation of stored disk media over time. 
> 
> 
> Not sure about that; DLT tapes are pretty bulky themselves; I think the
> difference between say a set of 4 DLT tapes and a single Maxtor 320 in
> caddy would be minimal. As for stored media, I think Maxtor are quoting
> 1M hours MTTF - (I hate to think how you measure such a figure) - for
> the 320G, and that is probably longer than I'd trust either the tape or
> the drive to survive.

i DO seriously doubt that this figure includes removing the drive, 
stuffing it in a siutcase or similar, loading on a truck/car/bike and 
unloading at a remote site.

Regards,
L.

-- 
Luca Berra -- bluca@comedia.it
	Service Delivery Manager
	Communication Media & Services S.r.l.
	Via A. Modigliani 1 - MILANO
	

Re: RAID backup

[Richard B. Johnson]

On Fri, 4 Oct 2002, Luca Berra wrote:

> Dr. David Alan Gilbert wrote:
> > * Alan Cox (alan@lxorguk.ukuu.org.uk) wrote:
> > 
> >>The problem with disks is you still have to archive them somewhere, and
> >>they are bulky. I also dont know what studies are available on the
> >>degradation of stored disk media over time. 
> > 
> > 
> > Not sure about that; DLT tapes are pretty bulky themselves; I think the
> > difference between say a set of 4 DLT tapes and a single Maxtor 320 in
> > caddy would be minimal. As for stored media, I think Maxtor are quoting
> > 1M hours MTTF - (I hate to think how you measure such a figure) - for
> > the 320G, and that is probably longer than I'd trust either the tape or
> > the drive to survive.
> 
> i DO seriously doubt that this figure includes removing the drive, 
> stuffing it in a siutcase or similar, loading on a truck/car/bike and 
> unloading at a remote site.
> 
> Regards,
> L.

Fire-wire 80 GB external drives work fine for this (Maxtor and others).
Keep them in a cool, not too dry (30 - 50 % RH), area. After a year or
two, they may take several power-cycles to start them up. You should
store them on an edge. This helps keep lube from weeping from the
sintered-bronze (Oilite) bearing, onto the platters.

Electronics that contain electrolytic capacitors shouldn't be stored
where the RH is below 10 % (either should tapes). The platters of
these disk-drives are in an environment where air-pressure can equalize
so they are not really "sealed". Instead, any air entering gets filtered.
Therefore, keep them away from things that leak nasty vapors like
batteries, paint, and fuels.

I've tried for years to find tape-drives that can reliably restore
backed-up data. Once you get to multi- Gigabyte drives, you can just
forget it. The data reliability necessary to read through 80 or more
gigibytes of data (serially) with no errors, to get to the file you need,
is just not available in any drive I've tested including the newest Sony
DLT.

The Linux SCSI tape driver will not attempt to read past a "permanent"
error. In principle, it could find the next tape-mark, then continue,
but it doesn't. Once an error occurs that the drive can't electronically
correct, that's all she wrote. It's not like a disk where you can just
tell some software layer to never read a bad "sector" again.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
The US military has given us many words, FUBAR, SNAFU, now ENRON.
Yes, top management were graduates of West Point and Annapolis.

Re: RAID backup

[Herman Oosthuysen]

Howdy,

The MTTF number is calculated during a process intended to identify 
weaknesses in the design of electronic equipment.  Depending on how you 
model the failure modes and who's failure tables you use, the MTTF 
number will vary enormously.

The important thing is not the number per se, but rather the quality 
review process associated with the calculation effort.

Unfortunately, the MTTF number became a marketing fad, with the result 
that some companies will calculate it, without doing any quality 
reviews, purely for marketing purposes.  There is no way to tell what 
they did.  The MTTF number is consequently totally meaningless by itself.

However, judging by my own experience, Maxtor drives are quite reliable 
and should last 3 years or more when in use and just about indefinitely 
when in storage.

Self demagnetization used to be a problem of magnetic media and some 
components such as capacitors used to deteriorate with age, but I think 
that those problems have been solved decades ago, so equipment in clean 
and dry storage should last almost forever.

The important thing to remember with disks and tapes is that they will 
eventually fail.  When that will happen is anybody's guess, but you have 
to plan for the eventuality; you can't just sit on your hands and hope 
for the best and the backup measures that you implement, should be 
commensurate with the value of the data.

Cheers,

Herman
http://www.AerospaceSoftware.com

Luca Berra wrote:
> Dr. David Alan Gilbert wrote:
> 
>> * Alan Cox (alan@lxorguk.ukuu.org.uk) wrote:
>>
>>> The problem with disks is you still have to archive them somewhere, and
>>> they are bulky. I also dont know what studies are available on the
>>> degradation of stored disk media over time. 
>>
>>
>>
>> Not sure about that; DLT tapes are pretty bulky themselves; I think the
>> difference between say a set of 4 DLT tapes and a single Maxtor 320 in
>> caddy would be minimal. As for stored media, I think Maxtor are quoting
>> 1M hours MTTF - (I hate to think how you measure such a figure) - for
>> the 320G, and that is probably longer than I'd trust either the tape or
>> the drive to survive.
> 
> 
> i DO seriously doubt that this figure includes removing the drive, 
> stuffing it in a siutcase or similar, loading on a truck/car/bike and 
> unloading at a remote site.
> 
> Regards,
> L.
> 

-- 

------------------------------------------------------------------------
Herman Oosthuysen
B.Eng.(E), Member of IEEE
Wireless Networks Inc.
http://www.WirelessNetworksInc.com
E-mail: Herman@WirelessNetworksInc.com
Phone: 1.403.569-5687, Fax: 1.403.235-3964
------------------------------------------------------------------------

Re: RAID backup

[Russell King]

On Fri, Oct 04, 2002 at 09:31:26AM -0600, Herman Oosthuysen wrote:
> Self demagnetization used to be a problem of magnetic media and some 
> components such as capacitors used to deteriorate with age, but I think 
> that those problems have been solved decades ago, so equipment in clean 
> and dry storage should last almost forever.

You missed "stable temperature" as well.  Some capacitors still
"dry out" with age and heat, even with todays technology.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: RAID backup

[Herman Oosthuysen]

Yep, a constant lowish temperature will also help a little, but I 
honestly think that in storage, a disk drive bearing will seize due to 
corrosion and lack of movement, before the electronics will fail.  You 
can prevent bearing (and capacitor) problems by starting the drive up 
every 6 months for a while though.

The effects of long term storage is experienced with military equipment, 
which frequently have to stay in storage for 40 or 50 years without ever 
really being used.  The stuff in storage has to be cycled through the 
workshops to keep them operational, which is part of the high cost of peace.

If disk drives are used for backup purposes, then I would suggest that 
they are rotated, so that they all remain in occational use, which 
should help to keep them alive.  Of course, the first thing to fail, 
would be the connector of the removable drive bay and there is nothing 
you can do about that, except to make sure that the connectors are gold 
plated to begin with.

I don't think that people have to worry too much about transportation 
vibration and shock to/from an off-site storage facility.  These things 
are quite rugged when not spinning, so if you transport them in 
styrofoam boxes and don't drop them on the floor, they should be OK.

BTW, putting a drive in storage for 40 or 50 years is not recommended 
for another reason: Obsolecense.  In 40 or 50 years, you probably won't 
have a computer that can use these drives anymore!  So, the only way to 
keep data long term, is to rotate the media continuously and upgrade as 
new technology is introduced.

Cheers,

Herman
http://www.AerospaceSoftware.com

Russell King wrote:
> On Fri, Oct 04, 2002 at 09:31:26AM -0600, Herman Oosthuysen wrote:
> 
>>Self demagnetization used to be a problem of magnetic media and some 
>>components such as capacitors used to deteriorate with age, but I think 
>>that those problems have been solved decades ago, so equipment in clean 
>>and dry storage should last almost forever.
> 
> 
> You missed "stable temperature" as well.  Some capacitors still
> "dry out" with age and heat, even with todays technology.
> 

-- 

------------------------------------------------------------------------
Herman Oosthuysen
B.Eng.(E), Member of IEEE
Wireless Networks Inc.
http://www.WirelessNetworksInc.com
E-mail: Herman@WirelessNetworksInc.com
Phone: 1.403.569-5687, Fax: 1.403.235-3964
------------------------------------------------------------------------

Re: RAID backup - media

[Alvin Oga]

hi ya alan

yuppers... 

those 320GB disks are not cheap?? but are floating around

- i use backups for "warm swap".... and dont care about it over time sorta
  thing ... need to be able to keep TBs of data online as fast as possible
  ( few minutes ) to restore a dead/hacked box..

- for "fast turn around"... tapes have always been dayz of effort to
  restore  and too much of a headache to keep the tapes and heads clean
  and test it more rigorously than having to test disks 

- you already implicitly trusts disks to hold your data .. till the disk
  dies ... hopefully due to ball bearing/lubricant/heat failure etc

fun stuff

have fun
alvin

On 4 Oct 2002, Alan Cox wrote:

> On Fri, 2002-10-04 at 12:20, Alvin Oga wrote:
> > we can build an 8-drive ( 120GB at $200ea )  or ( 160GB at $300>? each )
> > 1U box...about 0.960 - 1.28 TB each backup server ( 1U ) for under $2,500 in parts
> > +  cost of raid setup/testing is up to the user
> > 	- am thinking the 1.6TB of storage for 10K lira(?) is too much
> > 
> > i prefer disks to backup data.. so that its always a semi-warm backup
> > ( tapes have always been way tooo slow to find a file and to restore 
> 
> The problem with disks is you still have to archive them somewhere, and
> they are bulky. I also dont know what studies are available on the
> degradation of stored disk media over time. 
> 
> Capacity is not a problem, 3ware do a 12 channel sata card, with maxtor
> drives that comes in at 320x12 = 3.5Tb
> 

Re: RAID backup

[Kanoalani Withington]

Illtud Daniel wrote:

>Effrem Norwood wrote:
>
>>In addition, HSM software costs
>>something (even if you write it yourself) on top of the tape infrastructure.
>>One customer of ours was quoted 40K per TB of HSM *software* alone.
>>
>
>I've got 25TB uncompressed of HSM here, and it's cost us (ex VAT)
>roughly:
>
>£10k for the first 1.6TB (on NT, DLT library)
>£18k for the next  6.0TB (on NT, LTO library)
>£45k for the next 18.0TB (on Solaris, LTO Library)
>
>...for the software licencing alone. Plus about 10% pa. in support
>costs.
>You're looking at about 50-60% of the library cost for the HSM software
>to manage it (tapes are another thing). Is HSM really that difficult?
>
>It really is a racket, but it's not so much compared with the
>cost of re-producing the data (mainly digitized collections).
>I'd be happier about it if they were more reliable (libs and s/w).
>Disk arrays, on the other hand, would cost us a fortune in
>upgrading the cooling - we've had to do this once just because of
>the 3-4 TB of online storage we've got, and adding huge exchangers
>(and associated pipes) isn't something I want to do much of.
>
I agree it's a total racket. I've spent an appalling amount of money on 
this stuff over the years considering how simple it is. Last year I 
finally built mtx, the open source tape library driver, and wrote my own 
software in tcl scripts for a new archiving system. It really is that 
simple, I don't know how they can charge so much for thier software, 
especially when some it is junk to begin with.

-Kanoa

>
>
>Oh, and having spent much of last night and this morning dealing
>with multiple SCSI disk failures, and having seen about 5% of
>ours fail in a year, I'm rapidly seeing the light on IDE.
>


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID backup - mtx w/ tcl

[Alvin Oga]

hi ya kanoalani

are ya willing to release that mtx code ??
( well more like where can i find it )

i'd like to add it to the collection
	http://www.Linux-Backup.net/app.gwif.html
	
thanx
alvin

On Fri, 4 Oct 2002, Kanoalani Withington wrote:

> 
...

> I agree it's a total racket. I've spent an appalling amount of money on 
> this stuff over the years considering how simple it is. Last year I 
> finally built mtx, the open source tape library driver, and wrote my own 
> software in tcl scripts for a new archiving system. It really is that 
> simple, I don't know how they can charge so much for thier software, 
> especially when some it is junk to begin with.
> 
> -Kanoa
> 
> >

RE: RAID backup - mtx w/ tcl

[Effrem Norwood]

I was going to ask the same thing. That would be great.

Thanks,

Eff Norwood

> hi ya kanoalani
>
> are ya willing to release that mtx code ??
> ( well more like where can i find it )
>
> i'd like to add it to the collection
> 	http://www.Linux-Backup.net/app.gwif.html
>
> thanx
> alvin
>
> On Fri, 4 Oct 2002, Kanoalani Withington wrote:
>
> >
> ...
>
> > I agree it's a total racket. I've spent an appalling amount of money on
> > this stuff over the years considering how simple it is. Last year I
> > finally built mtx, the open source tape library driver, and
> wrote my own
> > software in tcl scripts for a new archiving system. It really is that
> > simple, I don't know how they can charge so much for thier software,
> > especially when some it is junk to begin with.
> >
> > -Kanoa
> >
> > >
>
>
>

Re: RAID backup - mtx w/ tcl

[Kanoalani Withington]

Hi Alvin,

Funny, when I searched for it on the web I didn't find anything. The 
original was written by Leonard Zubkoff and subsequently modified by 
Eric Lee Green. I never got it to work for me until around version 
1.2.15. I emailed Eric to see what's up with it but in the meantime I 
put my latest copy of it on my anonymous ftp server:

ftp.cfht.hawaii.edu:/pub/daprog/kanoa/mtx-1.2.16rel.tar.gz

Its got a GPL so that must be O.K.

It is great software. Basically it uses the scsi generic device to 
provide command line access to the device robotics so you can move 
tapes, use the export slots and barcode readers etc in a tape or optical 
jukebox.  It's a snap to then write archive and backup scripts that can 
pick and track tapes in the library.

When I first started writing scripts instead of buying $10k+ software 
packages I was using JBdriver, a commercial product that cost me nearly 
$2k (comparatively a good deal) which does the same thing. When they 
wanted another $1k to switch from a solaris box to a linux box I ditched 
it for mtx which works better and faster. It also installed in less than 
5min.

I wonder what happened to it?

-Kanoa

Alvin Oga wrote:

>
>hi ya kanoalani
>
>are ya willing to release that mtx code ??
>( well more like where can i find it )
>
>i'd like to add it to the collection
>	http://www.Linux-Backup.net/app.gwif.html
>	
>thanx
>alvin
>
>On Fri, 4 Oct 2002, Kanoalani Withington wrote:
>
>...
>
>>I agree it's a total racket. I've spent an appalling amount of money on 
>>this stuff over the years considering how simple it is. Last year I 
>>finally built mtx, the open source tape library driver, and wrote my own 
>>software in tcl scripts for a new archiving system. It really is that 
>>simple, I don't know how they can charge so much for thier software, 
>>especially when some it is junk to begin with.
>>
>>-Kanoa
>>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: RAID backup - mtx w/ tcl

[Luca Berra]

On Fri, Oct 04, 2002 at 12:22:03PM -1000, Kanoalani Withington wrote:
>Hi Alvin,
>
>Funny, when I searched for it on the web I didn't find anything. The 
>original was written by Leonard Zubkoff and subsequently modified by 
>Eric Lee Green. I never got it to work for me until around version 
>1.2.15. I emailed Eric to see what's up with it but in the meantime I 
>put my latest copy of it on my anonymous ftp server:
>
>ftp.cfht.hawaii.edu:/pub/daprog/kanoa/mtx-1.2.16rel.tar.gz
>
mtx is hosted on sourceforge http://sourceforge.net/projects/mtx
but i believe Alvin was referring to your tcl scripts

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: RAID backup - mtx w/ tcl

[Kanoalani Withington]

Ah, pardon me.

This may be off subject for this list. I'm only replying here because 
I've lost track of everyone on this thread but I think they were all 
here originally.

Unfortunately I don't have them nicely packaged so they can be used 
generically. I would be willing to do that if there was a need for it 
but I think there are plenty of good GPL archive/backup software 
packages. The only useful thing I think I have is a command line library 
interface which manages, names, tracks, loads and unloads tapes from a 
tape or CD/DVD library using a low level driver like mtx. It has a 
simple text interface and supports device locking and time stamps on 
library operations. It's what I use in all my archive/backup scripts to 
interface with media libraries.

Unfortunately it also uses an SQL database but I could make a standalone 
version if it would be useful to people.

I put what I have on my anonymous ftp server:

  ftp.cfht.hawaii.edu:/pub/daprog/kanoa/libh.tgz

I doubt it is useful to anyone in the state it is in though it might 
have some ideas that are useful and it could adapted to another 
installation.

Aloha,

-Kanoa

Luca Berra wrote:

> On Fri, Oct 04, 2002 at 12:22:03PM -1000, Kanoalani Withington wrote:
>
>> Hi Alvin,
>>
>> Funny, when I searched for it on the web I didn't find anything. The 
>> original was written by Leonard Zubkoff and subsequently modified by 
>> Eric Lee Green. I never got it to work for me until around version 
>> 1.2.15. I emailed Eric to see what's up with it but in the meantime I 
>> put my latest copy of it on my anonymous ftp server:
>>
>> ftp.cfht.hawaii.edu:/pub/daprog/kanoa/mtx-1.2.16rel.tar.gz
>>
> mtx is hosted on sourceforge http://sourceforge.net/projects/mtx
> but i believe Alvin was referring to your tcl scripts
>
> L.
>

Re: RAID backup - mtx w/ tcl

[Alvin Oga]

aloha kanoa

iz a kama-aina 2... or at least was... 20+yrs ago..

"usefulness" of an app is highly dependent on the reader??
	- think some coders look for one or two lines of
	"how did they do that" and merge that process/proceedure
	into their code

- think having a tcl based backup gui is useful

- my backup gui is html-based ( Backup.www.pl )
	- w/ search capability so that users can do their own
	searching in backups for the missing files they want restored

- collection of backup scripts ( backup to any media )
	http://www.Linux-Backup.net


layta bruddah ... ( sorry.. couldnt resist .. cracking up )
alvin


On Wed, 9 Oct 2002, Kanoalani Withington wrote:

> Ah, pardon me.
> 
> This may be off subject for this list. I'm only replying here because 
> I've lost track of everyone on this thread but I think they were all 
> here originally.
> 
> Unfortunately I don't have them nicely packaged so they can be used 
> generically. I would be willing to do that if there was a need for it 
> but I think there are plenty of good GPL archive/backup software 
> packages. The only useful thing I think I have is a command line library 
> interface which manages, names, tracks, loads and unloads tapes from a 
> tape or CD/DVD library using a low level driver like mtx. It has a 
> simple text interface and supports device locking and time stamps on 
> library operations. It's what I use in all my archive/backup scripts to 
> interface with media libraries.
> 
> Unfortunately it also uses an SQL database but I could make a standalone 
> version if it would be useful to people.
> 
> I put what I have on my anonymous ftp server:
> 
>   ftp.cfht.hawaii.edu:/pub/daprog/kanoa/libh.tgz
> 
> I doubt it is useful to anyone in the state it is in though it might 
> have some ideas that are useful and it could adapted to another 
> installation.

...

> >> ftp.cfht.hawaii.edu:/pub/daprog/kanoa/mtx-1.2.16rel.tar.gz
> >>
> > mtx is hosted on sourceforge http://sourceforge.net/projects/mtx
> > but i believe Alvin was referring to your tcl scripts
> >
> > L.
> >

Re: RAID backup

[Roy Sigurd Karlsbakk]

sorry

wrong answer
wrong mail
oops

On Thursday 03 October 2002 13:20, jbradford@dial.pipex.com wrote:
> Might it not be a good idea to DD the raw contents of each disk to a tape
> drive, just incase you fubar the array?  It would be time consuming, but at
> least you could restore your data in the event that it gets corrupted.
>
> John.

-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: RAID backup

[Adam Luter]

You would still have to mount -o remount,ro while doing this...

I'd just tape backup the actual files.  Then you can use that backup
on something other than the same raid cluster/disks.  Also, you don't
have to backup everything (such as /usr), unless you want to/can
afford to.

-Gryn (Adam Luter)

On Thu, Oct 03, 2002 at 12:20:21PM +0100, jbradford@dial.pipex.com wrote:
> Might it not be a good idea to DD the raw contents of each disk to a
tape drive, just incase you fubar the array?  It would be time
consuming, but at least you could restore your data in the event that
it gets corrupted.
> 
> John.