Re: Need help to recover root filesystem after a power supply issue

[Andrey Zhunev <a-j@a-j.ru>]

Ok, the ddrescue finished copying whatever it was able to recover.
There were many unreadable sectors near the end of the drive.
In total, there were over 170 pending sectors reported by SMART.

I then ran the following commands:

# smartctl -l scterc,900,100 /dev/sda
# echo 180 > /sys/block/sda/device/timeout

But this didn't help at all. The unreadable sectors still remained
unreadable.

So I wiped them with hdparm:
# hdparm --yes-i-know-what-i-am-doing --write-sector <sector_number> /dev/sda

I then re-read all these sectors, and they were all read correctly.

The number of pending sectors reported by SMART dropped down to 7.
Interestingly, there are still NO reallocated sectors reported.


Now, the xfs-repair reported the following:

# xfs_repair /dev/centos/root
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.


But I was still unable to momunt the filesystem:

# mount /dev/centos/root /tmp/root/
mount: mount /dev/mapper/centos-root on /tmp/root failed: Structure needs cleaning


So I went ahead with the '-L', and after some time the xfs_repair
completed the repair.


I can now mount my / partition successfully!
Most of the data seems to be there (I didn't check all of it, yet).

First thing I checked was the original kernel log. And yes, Chris was
right! There are a few warnings reporting read errors on /dev/sda.
These obviously were logged before the PSU failed. So the PSU might
have been just a coincidence...


Thanks a lot to everybody for your great help and support!!!



BTW, interestingly enough, those 7 pending sectors "disappeared" after
a power cycle. Maybe they were on some internal space on the HDD, not
accessible to users?
The SMART on this drive looks pretty clean now:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       867
  3 Spin_Up_Time            0x0027   181   179   021    Pre-fail  Always       -       5916
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       175
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       56949
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       175
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       112
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       62
194 Temperature_Celsius     0x0022   117   091   000    Old_age   Always       -       33
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0




---
Best regards,
 Andrey