ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Martin]

Help welcomed...


Trying to delete one device from a btrfs raid1 across 3 devices gives
for my example:


btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
device


That part of the filesystem is:

Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
        Total devices 3 FS bytes used 192.45GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5


All on: 3.14.10-gentoo


Before I do a mega-yank-up to the 3.18 kernel, is the above a known
problem or am I missing something?


(Device delete needed due to that device throwing sector errors :-( )

Thanks,
Martin

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[David Sterba]

On Mon, Mar 30, 2015 at 09:59:05PM +0100, Martin wrote:
> btrfs device delete /dev/sdf5 /mnt/data2
> 
> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
> device

Strange, that would mean that the ioctl number is not handled at all. I
haven't found any change in device delete on the userspace side. This
would mean that /mnt/data2 is not mounted. Did you try to do lazy
unmount?

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Martin]

On 31/03/15 18:37, David Sterba wrote:
> On Mon, Mar 30, 2015 at 09:59:05PM +0100, Martin wrote:
>> btrfs device delete /dev/sdf5 /mnt/data2
>>
>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
>> device
> 
> Strange, that would mean that the ioctl number is not handled at all. I
> haven't found any change in device delete on the userspace side. This
> would mean that /mnt/data2 is not mounted. Did you try to do lazy
> unmount?

Nope, that filesystem is still mounted and in use, but still a very good
deduction...

Trying smartmonctl just now on that drive (/dev/sdf) gives errors
suggesting that the drive is no longer responding at all :-(

There is also a steady repeat of:

kernel: sd 7:0:0:0: [sdf] CDB:
kernel: cdb[0]=0x2a: 2a 00 04 02 45 80 00 00 08 00
kernel: sd 7:0:0:0: [sdf] Unhandled error code
kernel: sd 7:0:0:0: [sdf]
kernel: Result: hostbyte=0x04 driverbyte=0x00

(Must watch that lot doesn't fill the log.)


Should I try a:

btrfs device delete missing /mnt/data2

or do I need to unmount or remount first?


... And the system is live and not to be rebooted if at all possible
until the weekend...

Thanks,
Martin

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Anand Jain]

>>> btrfs device delete /dev/sdf5 /mnt/data2
>>>
>>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
>>> device

  very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

- Anand

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Martin]

On 01/04/15 08:06, Anand Jain wrote:
> 
>>>> btrfs device delete /dev/sdf5 /mnt/data2
>>>>
>>>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
>>>> device
> 
>  very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

Looks like my /dev/sdf isn't responding with anything useful at all... A
firmware crash?... (This is for a 256GB SSD.)


# btrfs fi show -m

[...]

Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
        Total devices 3 FS bytes used 159.31GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5


btrfs-progs v3.19.1



# smartctl -i /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
(local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               /7:0:0:0
Product:
Compliance:           SPC-5
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50
bd_len=46
scsiModePageOffset: response length too short, resp_len=47 offset=50
bd_len=46
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more
'-T permissive' options.



And btrfs is still running ok so far... I'll be swapping that device at
the weekend. (First chance I have for a Sunday shutdown. :-( )


Still... For administering btrfs, it is a little disturbing not to be
able to remove/delete a device unless that device is both mounted and
working...

Or is that where the "missing" option comes in?



Comments welcomed,

Thanks,
Martin

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Anand Jain]

  Looks like an option to use devid to delete a device would
  have mitigated the issue.  Also the error reported is no
  where near the reality. Will fix them.

  Thanks for reporting.

Anand


On 04/01/2015 06:54 PM, Martin wrote:
> On 01/04/15 08:06, Anand Jain wrote:
>>
>>>>> btrfs device delete /dev/sdf5 /mnt/data2
>>>>>
>>>>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
>>>>> device
>>
>>   very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.
>
> Looks like my /dev/sdf isn't responding with anything useful at all... A
> firmware crash?... (This is for a 256GB SSD.)
>
>
> # btrfs fi show -m
>
> [...]
>
> Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
>          Total devices 3 FS bytes used 159.31GiB
>          devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
>          devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
>          devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5
>
>
> btrfs-progs v3.19.1
>
>
>
> # smartctl -i /dev/sdf
> smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
> (local build)
> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Vendor:               /7:0:0:0
> Product:
> Compliance:           SPC-5
> User Capacity:        600,332,565,813,390,450 bytes [600 PB]
> Logical block size:   774843950 bytes
> scsiModePageOffset: response length too short, resp_len=47 offset=50
> bd_len=46
> scsiModePageOffset: response length too short, resp_len=47 offset=50
> bd_len=46
>>> Terminate command early due to bad response to IEC mode page
> A mandatory SMART command failed: exiting. To continue, add one or more
> '-T permissive' options.
>
>
>
> And btrfs is still running ok so far... I'll be swapping that device at
> the weekend. (First chance I have for a Sunday shutdown. :-( )
>
>
> Still... For administering btrfs, it is a little disturbing not to be
> able to remove/delete a device unless that device is both mounted and
> working...
>
> Or is that where the "missing" option comes in?
>
>
>
> Comments welcomed,
>
> Thanks,
> Martin

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Martin]

Anand,

Thanks for picking up.

The devid option sounds good.


Thanks,
Martin



On 01/04/15 14:15, Anand Jain wrote:
> 
> 
>  Looks like an option to use devid to delete a device would
>  have mitigated the issue.  Also the error reported is no
>  where near the reality. Will fix them.
> 
>  Thanks for reporting.
> 
> Anand
> 
> 
> On 04/01/2015 06:54 PM, Martin wrote:
>> On 01/04/15 08:06, Anand Jain wrote:
>>>
>>>>>> btrfs device delete /dev/sdf5 /mnt/data2
>>>>>>
>>>>>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl
>>>>>> for
>>>>>> device
>>>
>>>   very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.
>>
>> Looks like my /dev/sdf isn't responding with anything useful at all... A
>> firmware crash?... (This is for a 256GB SSD.)
>>
>>
>> # btrfs fi show -m
>>
>> [...]
>>
>> Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
>>          Total devices 3 FS bytes used 159.31GiB
>>          devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
>>          devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
>>          devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5
>>
>>
>> btrfs-progs v3.19.1
>>
>>
>>
>> # smartctl -i /dev/sdf
>> smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
>> (local build)
>> Copyright (C) 2002-14, Bruce Allen, Christian Franke,
>> www.smartmontools.org
>>
>> === START OF INFORMATION SECTION ===
>> Vendor:               /7:0:0:0
>> Product:
>> Compliance:           SPC-5
>> User Capacity:        600,332,565,813,390,450 bytes [600 PB]
>> Logical block size:   774843950 bytes
>> scsiModePageOffset: response length too short, resp_len=47 offset=50
>> bd_len=46
>> scsiModePageOffset: response length too short, resp_len=47 offset=50
>> bd_len=46
>>>> Terminate command early due to bad response to IEC mode page
>> A mandatory SMART command failed: exiting. To continue, add one or more
>> '-T permissive' options.
>>
>>
>>
>> And btrfs is still running ok so far... I'll be swapping that device at
>> the weekend. (First chance I have for a Sunday shutdown. :-( )
>>
>>
>> Still... For administering btrfs, it is a little disturbing not to be
>> able to remove/delete a device unless that device is both mounted and
>> working...
>>
>> Or is that where the "missing" option comes in?
>>
>>
>>
>> Comments welcomed,
>>
>> Thanks,
>> Martin
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Anand Jain]

Martin,

  presumably kernel failed to read SB from disk /dev/sdf5,
  bit strange it saw 'Inappropriate ioctl for device',
  logically that don't explain.

  Curios to know what does 'btrfs fi show -d' say ?
  and output of blkid if you could. The situation here
  is not same as destroying the SB though.

Thanks, Anand


On 02/04/2015 01:28, Martin wrote:
> Anand,
>
> Thanks for picking up.
>
> The devid option sounds good.
>
>
> Thanks,
> Martin
>
>
>
> On 01/04/15 14:15, Anand Jain wrote:
>>
>>
>>   Looks like an option to use devid to delete a device would
>>   have mitigated the issue.  Also the error reported is no
>>   where near the reality. Will fix them.
>>
>>   Thanks for reporting.
>>
>> Anand
>>
>>
>> On 04/01/2015 06:54 PM, Martin wrote:
>>> On 01/04/15 08:06, Anand Jain wrote:
>>>>
>>>>>>> btrfs device delete /dev/sdf5 /mnt/data2
>>>>>>>
>>>>>>> ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl
>>>>>>> for
>>>>>>> device
>>>>
>>>>    very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.
>>>
>>> Looks like my /dev/sdf isn't responding with anything useful at all... A
>>> firmware crash?... (This is for a 256GB SSD.)
>>>
>>>
>>> # btrfs fi show -m
>>>
>>> [...]
>>>
>>> Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
>>>           Total devices 3 FS bytes used 159.31GiB
>>>           devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
>>>           devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
>>>           devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5
>>>
>>>
>>> btrfs-progs v3.19.1
>>>
>>>
>>>
>>> # smartctl -i /dev/sdf
>>> smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
>>> (local build)
>>> Copyright (C) 2002-14, Bruce Allen, Christian Franke,
>>> www.smartmontools.org
>>>
>>> === START OF INFORMATION SECTION ===
>>> Vendor:               /7:0:0:0
>>> Product:
>>> Compliance:           SPC-5
>>> User Capacity:        600,332,565,813,390,450 bytes [600 PB]
>>> Logical block size:   774843950 bytes
>>> scsiModePageOffset: response length too short, resp_len=47 offset=50
>>> bd_len=46
>>> scsiModePageOffset: response length too short, resp_len=47 offset=50
>>> bd_len=46
>>>>> Terminate command early due to bad response to IEC mode page
>>> A mandatory SMART command failed: exiting. To continue, add one or more
>>> '-T permissive' options.
>>>
>>>
>>>
>>> And btrfs is still running ok so far... I'll be swapping that device at
>>> the weekend. (First chance I have for a Sunday shutdown. :-( )
>>>
>>>
>>> Still... For administering btrfs, it is a little disturbing not to be
>>> able to remove/delete a device unless that device is both mounted and
>>> working...
>>>
>>> Or is that where the "missing" option comes in?
>>>
>>>
>>>
>>> Comments welcomed,
>>>
>>> Thanks,
>>> Martin
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Martin]

On 06/04/15 14:32, Anand Jain wrote:
> btrfs fi show -d


That gives:

# btrfs fi show -d
warning, device 3 is missing
warning devid 3 not found already
warning, device 3 is missing
warning devid 3 not found already
Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
        Total devices 3 FS bytes used 30.94GiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data'  uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223
        Total devices 3 FS bytes used 95.74GiB
        devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
        devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
        devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2'  uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
        Total devices 3 FS bytes used 151.60MiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
        *** Some devices missing

Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
        Total devices 3 FS bytes used 159.34GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        *** Some devices missing

btrfs-progs v3.19.1


And without the "-d":

# btrfs fi show
Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
        Total devices 3 FS bytes used 30.94GiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data'  uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223
        Total devices 3 FS bytes used 95.74GiB
        devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
        devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
        devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2'  uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
        Total devices 3 FS bytes used 151.60MiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sdf4

Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
        Total devices 3 FS bytes used 159.34GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1


Interestingly, all the log messages about /dev/sdf are now no longer
being repeated.

(And nope, not had a chance to swap that disk yet!)


Hence, should I do a "btrfs device delete missing /mnt/data2"?

Cheers,
Martin

Re: ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

[Anand Jain]

Thanks Martin for helping with the data.

Unfortunately no matter what I try I couldn't reproduce the
"Inappropriate ioctl for device"

But I got INVALD which is wrong as well. So I sent a patch for
that. EIO is appropriate in this case.


Next, Passing a devid to delete a device - yes it would help
patch is ready in my WS BUT - just noticed that error handling
code nicely put the FS into readonly when there is commit
failure so that's something which should be fixed along with
this patch which I am attempting. Since as in your case with
3 disks raid1, your reconstructed radi1 will still be considered
as healthy after taking out a disk.

Further, to recover from your situation you could try replace,
that will work. OR if you are not planning to add another disk then,
try this: (sorry needs a down time)

umount
remove failed disk from the system
mount -o degrade
btrfs dev del missing <mnt>
mount -o remount
(I hope you would check / mange the space availably part)


(By the way, those missing messages are user land fabricated,
  btrfs-progs: 206efb60cbe3049e0d44c6da3c1909aeee18f813

so don't depend on that, to know what kernel knows
probably you need /proc/fs/btrfs/devlist or sysfs
posted in the ML before,

btrfs fi show -m

was written so to know actual kernel visibility but
it was again crippled by above commit id.

David, should back out 206efb60cbe3049e0d44c6da3c1909aeee18f813
as mentioned before it will help.


On 04/07/2015 07:41 PM, Martin wrote:
> On 06/04/15 14:32, Anand Jain wrote:
>> btrfs fi show -d
>
>
> That gives:
>
> # btrfs fi show -d
> warning, device 3 is missing
> warning devid 3 not found already
> warning, device 3 is missing
> warning devid 3 not found already

David,

  As commented before - you shouldn't have integrated the patch

   915902c5002485fb13d27c4b699a73fb66cc0f09


Thanks,  Anand




> Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
>          Total devices 3 FS bytes used 30.94GiB
>          devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
>          devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
>          devid    3 size 24.00GiB used 24.00GiB path /dev/sde4
>
> Label: 'btrfs_data'  uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223
>          Total devices 3 FS bytes used 95.74GiB
>          devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
>          devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
>          devid    3 size 87.24GiB used 87.24GiB path /dev/sde5
>
> Label: 'btrfs_root2'  uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
>          Total devices 3 FS bytes used 151.60MiB
>          devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
>          devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
>          *** Some devices missing
>
> Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
>          Total devices 3 FS bytes used 159.34GiB
>          devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
>          devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
>          *** Some devices missing
>
> btrfs-progs v3.19.1
>
>
> And without the "-d":
>
> # btrfs fi show
> Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
>          Total devices 3 FS bytes used 30.94GiB
>          devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
>          devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
>          devid    3 size 24.00GiB used 24.00GiB path /dev/sde4
>
> Label: 'btrfs_data'  uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223
>          Total devices 3 FS bytes used 95.74GiB
>          devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
>          devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
>          devid    3 size 87.24GiB used 87.24GiB path /dev/sde5
>
> Label: 'btrfs_root2'  uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
>          Total devices 3 FS bytes used 151.60MiB
>          devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
>          devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
>          devid    3 size 24.00GiB used 24.00GiB path /dev/sdf4
>
> Label: 'btrfs_data2'  uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
>          Total devices 3 FS bytes used 159.34GiB
>          devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
>          devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
>          devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5
>
> btrfs-progs v3.19.1
>
>
> Interestingly, all the log messages about /dev/sdf are now no longer
> being repeated.
>
> (And nope, not had a chance to swap that disk yet!)
>
>
> Hence, should I do a "btrfs device delete missing /mnt/data2"?
>
> Cheers,
> Martin
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>