Re: multi-device btrfs with single data mode and disk failure

[Alexandre Poux <pums974@gmail.com>]

Le 20/09/2016 à 00:05, Alexandre Poux a écrit :
>
> Le 15/09/2016 à 23:54, Chris Murphy a écrit :
>> On Thu, Sep 15, 2016 at 3:48 PM, Alexandre Poux <pums974@gmail.com> wrote:
>>> Le 15/09/2016 à 18:54, Chris Murphy a écrit :
>>>> On Thu, Sep 15, 2016 at 10:30 AM, Alexandre Poux <pums974@gmail.com> wrote:
>>>>> Thank you very much for your answers
>>>>>
>>>>> Le 15/09/2016 à 17:38, Chris Murphy a écrit :
>>>>>> On Thu, Sep 15, 2016 at 1:44 AM, Alexandre Poux <pums974@gmail.com> wrote:
>>>>>>>     Is it possible to do some king of a "btrfs delete missing" on this
>>>>>>>     kind of setup, in order to recover access in rw to my other data, or
>>>>>>>     I must copy all my data on a new partition
>>>>>> That *should* work :) Except that your file system with 6 drives is
>>>>>> too full to be shrunk to 5 drives. Btrfs will either refuse, or get
>>>>>> confused, about how to shrink a nearly full 6 drive volume into 5.
>>>>>>
>>>>>> So you'll have to do one of three things:
>>>>>>
>>>>>> 1. Add a 2+TB drive, then remove the missing one; OR
>>>>>> 2. btrfs replace is faster and is raid10 reliable; OR
>>>>>> 3. Read only scrub to get a file listing of bad files, then remount
>>>>>> read-write degraded and delete them all. Now you maybe can do a device
>>>>>> delete missing. But it's still a tight fit, it basically has to
>>>>>> balance things out to get it to fit on an odd number of drives, it may
>>>>>> actually not work even though there seems to be enough total space,
>>>>>> there has to be enough space on FOUR drives.
>>>>>>
>>>>> Are you sure you are talking about data in single mode ?
>>>>> I don't understand why you are talking about raid10,
>>>>> or the fact that it will have to rebalance everything.
>>>> Yeah sorry I got confused in that very last sentence. Single, it will
>>>> find space in 1GiB increments. Of course this fails because that data
>>>> doesn't exist anymore, but to start the operation it needs to be
>>>> possible.
>>> No problem
>>>>> Moreover, even in degraded mode I cannot mount it in rw
>>>>> It tells me
>>>>> "too many missing devices, writeable remount is not allowed"
>>>>> due to the fact I'm in single mode.
>>>> Oh you're in that trap. Well now you're stuck. I've had the case where
>>>> I could mount read write degraded with metadata raid1 and data single,
>>>> but it was good for only one mount and then I get the same message you
>>>> get and it was only possible to mount read only. At that point it's
>>>> totally suck unless you're adept at manipulating the file system with
>>>> a hex editor...
>>>>
>>>> Someone might have a patch somewhere that drops this check and lets
>>>> too many missing devices to mount anyway... I seem to recall this.
>>>> It'd be in the archives if it exists.
>>>>
>>>>
>>>>
>>>>> And as far as as know, btrfs replace and btrfs delete, are not supposed
>>>>> to work in read only...
>>>> It doesn't. Must be read write mounted.
>>>>
>>>>
>>>>> I would like to tell him forgot about the missing data, and give me back
>>>>> my partition.
>>>> This feature doesn't exist yet. I really want to see this, it'd be
>>>> great for ceph and gluster if the volume could lose a drive, report
>>>> all the missing files to the cluster file system, delete the device
>>>> and the file references, and then the cluster knows that brick doesn't
>>>> have those files and can replicate them somewhere else or even back to
>>>> the brick that had them.
>>>>
>>> So I found this patch : https://patchwork.kernel.org/patch/7014141/
>>>
>>> Does this seems ok ?
>> No idea I haven't tried it.
>>
>>> So after patching my kernel with it,
>>> I should be able to mount in rw my partition, and thus,
>>> I will be able to do a btrfs delete missing
>>> Which will just forgot about the old disk and everything should be fine
>>> afterward ?
>> It will forget about the old disk but it will try to migrate all
>> metadata and data that was on that disk to the remaining drives; so
>> until you delete all files that are corrupt, you'll continue to get
>> corruption messages about them.
>>
>>> Is this risky ? or not so much ?
>> Probably. If you care about the data, mount read only, back up what
>> you can, then see if you can fix it after that.
>>
>>> The scrubing is almost finished, and as I was expecting, I lost no data
>>> at all.
>> Well I'd guess the device delete should work then, but I still have no
>> idea if that patch will let you mount it degraded read-write. Worth a
>> shot though, it'll save time.
>>
> OK, so I found some time to work on it.
>
> I decided to do some tests in a vm (virtualbox) with 3 disks
> after making an array with 3 disks, metadata in raid1 and data in single,
> I remove one disk to reproduce my situation.
>
> I tried the patch, and, after updated it (nothing fancy),
> I can indeed mount a degraded partition with data in single.
>
> But I can't remove the device :
> #btrfs device remove missing /mnt
> ERROR: error removing device 'missing': Input/output error
> or
> #btrfs device remove 2 /mnt
> ERROR: error removing devid 2: Input/output error
>
> replace doesn't work either
> btrfs replace start -B 2 /dev/sdb /mnt
> BTRFS error (device sda2): btrfs_scrub_dev(<missing disk>, 2, /dev/sdb)
> failed -12
> ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt": Cannot allocate
> memory, <illegal result value>
>
> but scrub works, and I can add and remove an another device if I want.
> It's just the missing device that I cant get rid of.
>
> Any other idea ?
If I wanted to try to edit my partitions with an hex editor, where would
I find infos on how to do that ?
I really don't want to go this way, but if this is relatively simple, it
may be worth to try.