device failure hangs the system

device failure hangs the system

[Anand Jain]

hi,

Following test case causes my remote system to hard-hang and does
not respond to any key strokes.

-----------
# btrfs fi show
failed to read /dev/sr0
Label: none  uuid: 75ad3c9f-f661-498e-8c13-89d4e4c58312
	Total devices 3 FS bytes used 28.00KB
	devid    1 size 465.76GB used 2.02GB path /dev/sdb
	devid    3 size 465.76GB used 2.01GB path /dev/sdd
	devid    2 size 465.76GB used 1.01GB path /dev/sdc

Btrfs v0.19-35-g1b444cd
------------


Stopping the disk
------------
# echo 1 > /sys/block/sdd/device/delete
------------
------------
# dmesg | tail
sd 3:0:0:0: [sdd] Stopping disk
::
#
------------

------------
# btrfs fi show
failed to read /dev/sr0
Label: none  uuid: 75ad3c9f-f661-498e-8c13-89d4e4c58312
	Total devices 3 FS bytes used 28.00KB
	devid    1 size 465.76GB used 2.02GB path /dev/sdb
	devid    2 size 465.76GB used 1.01GB path /dev/sdc
	*** Some devices missing

Btrfs v0.19-35-g1b444cd
------------

and the following command hangs the system.
-------------
# btrfs fi balance /btrfs
-------------

Any idea if this is an known issue ? OR if there is any better way
to fail a disk (or a loop disk) for testing.

(something similar to cfgadm -c unconfigure in solaris).

Thanks
-Anand

Re: device failure hangs the system

[Anand Jain]

  Looks like there is a panic (not system hang).
  any idea where is the panic log after the system has been
  power-recycled. (its not in the /var/log/messages or dmesg or
  /var/crash is empty)

Thanks, Anand


On 07/01/2011 05:25 PM, Anand Jain wrote:
>
> hi,
>
> Following test case causes my remote system to hard-hang and does
> not respond to any key strokes.
>
> -----------
> # btrfs fi show
> failed to read /dev/sr0
> Label: none uuid: 75ad3c9f-f661-498e-8c13-89d4e4c58312
> Total devices 3 FS bytes used 28.00KB
> devid 1 size 465.76GB used 2.02GB path /dev/sdb
> devid 3 size 465.76GB used 2.01GB path /dev/sdd
> devid 2 size 465.76GB used 1.01GB path /dev/sdc
>
> Btrfs v0.19-35-g1b444cd
> ------------
>
>
> Stopping the disk
> ------------
> # echo 1 > /sys/block/sdd/device/delete
> ------------
> ------------
> # dmesg | tail
> sd 3:0:0:0: [sdd] Stopping disk
> ::
> #
> ------------
>
> ------------
> # btrfs fi show
> failed to read /dev/sr0
> Label: none uuid: 75ad3c9f-f661-498e-8c13-89d4e4c58312
> Total devices 3 FS bytes used 28.00KB
> devid 1 size 465.76GB used 2.02GB path /dev/sdb
> devid 2 size 465.76GB used 1.01GB path /dev/sdc
> *** Some devices missing
>
> Btrfs v0.19-35-g1b444cd
> ------------
>
> and the following command hangs the system.
> -------------
> # btrfs fi balance /btrfs
> -------------
>
> Any idea if this is an known issue ? OR if there is any better way
> to fail a disk (or a loop disk) for testing.
>
> (something similar to cfgadm -c unconfigure in solaris).
>
> Thanks
> -Anand
>

Re: device failure hangs the system

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

On Fri, 01 Jul 2011 17:42:33 +0800
Anand Jain <Anand.Jain@oracle.com> wrote:

>   Looks like there is a panic (not system hang).
>   any idea where is the panic log after the system has been
>   power-recycled. (its not in the /var/log/messages or dmesg or
>   /var/crash is empty)

You can get it by seting up netconsole logging to another machine and then reproducing the crash.
http://www.cyberciti.biz/tips/linux-netconsole-log-management-tutorial.html

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: device failure hangs the system

[Josef Bacik]

On 07/01/2011 05:25 AM, Anand Jain wrote:
> 
> hi,
> 
> Following test case causes my remote system to hard-hang and does
> not respond to any key strokes.
> 
> -----------
> # btrfs fi show
> failed to read /dev/sr0
> Label: none  uuid: 75ad3c9f-f661-498e-8c13-89d4e4c58312
>     Total devices 3 FS bytes used 28.00KB
>     devid    1 size 465.76GB used 2.02GB path /dev/sdb
>     devid    3 size 465.76GB used 2.01GB path /dev/sdd
>     devid    2 size 465.76GB used 1.01GB path /dev/sdc
> 
> Btrfs v0.19-35-g1b444cd
> ------------
> 
> 
> Stopping the disk
> ------------
> # echo 1 > /sys/block/sdd/device/delete
> ------------

Well that's a neat trick, do you have a way to undo that action too?
Seems a rescan doesn't make it show back up.  Please try the patch I
just posted to the list to fix this problem.  Thanks,

Josef

Re: device failure hangs the system

[Anand Jain]

Josef,


> Well that's a neat trick, do you have a way to undo that action too?
> Seems a rescan doesn't make it show back up.


hope the following helps..
-------------
  # fdisk -l /dev/sdg | egrep "Disk /"
  Disk /dev/sdg: 4294 MB, 4294967296 bytes

  # x=`ls -l /sys/class/block/sdg | cut -d "/" -f12 | sed 's/:/ /g'`
  # echo "scsi remove-single-device ${x}" > /proc/scsi/scsi

  # fdisk -l /dev/sdg | egrep "Disk /"

  # echo "scsi add-single-device ${x}" > /proc/scsi/scsi

  # fdisk -l /dev/sdg | egrep "Disk /"
  Disk /dev/sdg: 4294 MB, 4294967296 bytes
-------------


>  Please try the patch I just posted to the list to fix this problem.  Thanks,

  Facing some challenges to upgrade my machine to 3.0.0-rc6, so is
  the delay.

  Thanks for the patch.

Anand

> Josef