* Re: btrfs-tools: missing device delete/remove cancel option on disk failure
2016-05-08 11:53 ` g6094199
@ 2016-05-08 15:35 ` Chris Murphy
2016-05-08 21:37 ` Duncan
1 sibling, 0 replies; 5+ messages in thread
From: Chris Murphy @ 2016-05-08 15:35 UTC (permalink / raw)
To: g6094199; +Cc: Btrfs BTRFS
On Sun, May 8, 2016 at 5:53 AM, <g6094199@freenet.de> wrote:
>
> i guess this log is out of diskussion:
>
> [44388.089321] sd 8:0:0:0: [sdf] tag#0 FAILED Result:
> hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
> [44388.089334] sd 8:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 00 43 1c 48
> 00 00 08 00
> [44388.089340] blk_update_request: I/O error, dev sdf, sector 35185216
>
> ...
>
> May 7 06:39:31 NAS-Sash kernel: [35777.520490] sd 8:0:0:0: [sdf] tag#0
> FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> May 7 06:39:31 NAS-Sash kernel: [35777.520500] sd 8:0:0:0: [sdf] tag#0
> Sense Key : Medium Error [current]
> May 7 06:39:31 NAS-Sash kernel: [35777.520508] sd 8:0:0:0: [sdf] tag#0
> Add. Sense: Unrecovered read error
> May 7 06:39:31 NAS-Sash kernel: [35777.520516] sd 8:0:0:0: [sdf] tag#0
> CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00
> May 7 06:39:31 NAS-Sash kernel: [35777.520522] blk_update_request:
> critical medium error, dev sdf, sector 472347008
> May 7 06:39:35 NAS-Sash kernel: [35781.364117] sd 8:0:0:0: [sdf] tag#0
> FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> May 7 06:39:35 NAS-Sash kernel: [35781.364138] sd 8:0:0:0: [sdf] tag#0
> Sense Key : Medium Error [current]
> May 7 06:39:35 NAS-Sash kernel: [35781.364146] sd 8:0:0:0: [sdf] tag#0
> Add. Sense: Unrecovered read error
> May 7 06:39:35 NAS-Sash kernel: [35781.364154] sd 8:0:0:0: [sdf] tag#0
> CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00
What drive is this? Is it one of the new drives? It's very unusual
that this new drive which should be getting mostly writes (what is
there to read?) during a 'dev delete' or 'replace' operation, but in
particular to have a read error this soon after data has been written.
I suspect maybe it's some other drive.
>
> and different vendors use the raw error rate differently. some count up
> constantly, some do only log real destructive errors.
>
> but i had the luck that the system froze completely. not even an log
> entry. now the file system is broken.....arg!
>
> now i need some advice what to do next....best practice wise? try to
> mount degraded and copy off all data? then i will net at least 9TB of
> new storage... :-(
If you don't have a backup already, you're kinda twice behind best
practices, aren't you? One you should have one anyway, two you should
have one before doing anything risky like growing the array or doing
device replacements.
The best practice at this point would be to mount with -o degraded,ro
I wouldn't even do rw mounts. The less you change things the better
off the file system is. And then copy the data to a new array before
making any changes to the original volume. And hopefully mounting
degraded,ro works, if not then it's btrfs restore (offline scrape)
time.
--
Chris Murphy
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: btrfs-tools: missing device delete/remove cancel option on disk failure
2016-05-08 11:53 ` g6094199
2016-05-08 15:35 ` Chris Murphy
@ 2016-05-08 21:37 ` Duncan
1 sibling, 0 replies; 5+ messages in thread
From: Duncan @ 2016-05-08 21:37 UTC (permalink / raw)
To: linux-btrfs
g6094199 posted on Sun, 08 May 2016 13:53:41 +0200 as excerpted:
> now i need some advice what to do next....best practice wise? try to
> mount degraded and copy off all data? then i will net at least 9TB of
> new storage... :-(
Well, even in general, regardless of the filesystem or what it's stored
on, the sysadmin's rule of backups, in simple form, says if it's not
backed up to at least one level, you are by your lack of backup defining
the data as worth less than the time, trouble and resources necessary to
do that backup.
And btrfs, being still stabilizing, not yet fully stable, reinforces that
rule. I know nobody around here that would recommend usage of btrfs in
any mode without backups, unless the data on it is truly of trivial
value, testing only and not worth backing up.
And btrfs parity-raid, raid56 mode, which you're using as you reported
raid5 (I'm assuming btrfs raid5 here), is new and not yet as stable as
btrfs is in general, with at least two known bugs related to recovery
from loss of device, making it for purposes of recovery planning raid0.
Nobody I know is even recommending that for production use even with
backups at this point. It's recommended only for testing, not for in-use
deployment unless you really /are/ prepared to be scrapping it and
starting over more or less constantly.
And of course the only way you couldn't know that before starting is if
you didn't care enough about the stability of your data to do your
research on the filesystem you were choosing in the first place, which
would again point to the data simply not being worth enough to care about.
Which means if that data wasn't already backed up, you were already
defining it as trivial testing data, and the only reason you may have for
trying to recover it is to test the recovery procedures. Otherwise,
consider it a loss and blow it away with a new badblocks or mkfs, ready
for a new test, as you were already declaring the data as testing data,
not worth trying to recover, by storing it on btrfs parity raid, which is
known to be recommended only for testing with already backed up or
trivial value data you plan on losing, at this point.
So you don't need 9 TB of new storage, because either you already have it
available and are using it for the backup you already have, or you had
already demonstrated in multiple different ways that the data was really
of only trivial value to you, so you can simply blow away the existing
broken filesystem and simply start over with a new test. Words can claim
otherwise, but actions don't lie. 8^0
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 5+ messages in thread