btrfs convert problem

btrfs convert problem

[Serhat Sevki Dincer]

Hi,

I have an external usb HDD (WD my passport, just usb cable, no
external power) with a single ext4 partition occupying the whole disk
with 698 GiB capacity and 188 GiB empty space. The data on disk is not
very important.

I also have a laptop with Manjaro 64-bit XFCE, kernel 4.14.68,
btrfs-progs v4.17.1, all packages come with Manjaro.

I tried to convert my disk to btrfs with
sudo btrfs-convert /dev/sdb1
I have also tried -i, -n options, all failed with:

create btrfs filesystem:
    blocksize: 4096
    nodesize:  16384
    features:  extref, skinny-metadata (default)
creating ext2 image file
ERROR: failed to create ext2_saved/image: -1
WARNING: an error occurred during conversion, filesystem is partially
created but not finalized and not mountable

I could not find this error/warning combo on the net.
ext4 partition seems intact and working after these attempts.
How can I debug and complete this conversion?

Thanks..

Re: btrfs convert problem

[Nikolay Borisov]

On 13.09.2018 15:22, Serhat Sevki Dincer wrote:
> Hi,
> 
> I have an external usb HDD (WD my passport, just usb cable, no
> external power) with a single ext4 partition occupying the whole disk
> with 698 GiB capacity and 188 GiB empty space. The data on disk is not
> very important.
> 
> I also have a laptop with Manjaro 64-bit XFCE, kernel 4.14.68,
> btrfs-progs v4.17.1, all packages come with Manjaro.
> 
> I tried to convert my disk to btrfs with
> sudo btrfs-convert /dev/sdb1
> I have also tried -i, -n options, all failed with:
> 
> create btrfs filesystem:
>     blocksize: 4096
>     nodesize:  16384
>     features:  extref, skinny-metadata (default)
> creating ext2 image file
> ERROR: failed to create ext2_saved/image: -1

-1 seems to be EPERM, is your device write-protected, readonly or
something like that ?

> WARNING: an error occurred during conversion, filesystem is partially
> created but not finalized and not mountable
> 
> I could not find this error/warning combo on the net.
> ext4 partition seems intact and working after these attempts.
> How can I debug and complete this conversion?
> 
> Thanks..
> 

Re: btrfs convert problem

[Serhat Sevki Dincer]

> -1 seems to be EPERM, is your device write-protected, readonly or
> something like that ?

I am able to use ext4 partition read/write, created a file and wrote
in it, un/re mounted it, all is ok.
The drive only has a microusb port and a little led light; no rw
switch or anything like that..

Re: btrfs convert problem

[Nikolay Borisov]

On 13.09.2018 19:15, Serhat Sevki Dincer wrote:
>> -1 seems to be EPERM, is your device write-protected, readonly or
>> something like that ?
> 
> I am able to use ext4 partition read/write, created a file and wrote
> in it, un/re mounted it, all is ok.
> The drive only has a microusb port and a little led light; no rw
> switch or anything like that..
> 

What might help is running btrfs convert under strace. So something like :

sudo strace -f -o strace.log btrfs-convert /dev/sdb1 and then send the log.

Re: btrfs convert problem

[Qu Wenruo]

On 2018/9/14 上午12:37, Nikolay Borisov wrote:
> 
> 
> On 13.09.2018 19:15, Serhat Sevki Dincer wrote:
>>> -1 seems to be EPERM, is your device write-protected, readonly or
>>> something like that ?
>>
>> I am able to use ext4 partition read/write, created a file and wrote
>> in it, un/re mounted it, all is ok.
>> The drive only has a microusb port and a little led light; no rw
>> switch or anything like that..
>>
> 
> What might help is running btrfs convert under strace. So something like :
> 
> sudo strace -f -o strace.log btrfs-convert /dev/sdb1 and then send the log.
> 
strace would greatly help in this case.

My guess is something wrong happened in migrate_super_block(), which
doesn't handle ret > 0 case well.
In that case, it means pwrite 4K doesn't finish in one call, which looks
pretty strange.

Thanks,
Qu

Re: btrfs convert problem

[Nikolay Borisov]

On 14.09.2018 02:17, Qu Wenruo wrote:
> 
> 
> On 2018/9/14 上午12:37, Nikolay Borisov wrote:
>>
>>
>> On 13.09.2018 19:15, Serhat Sevki Dincer wrote:
>>>> -1 seems to be EPERM, is your device write-protected, readonly or
>>>> something like that ?
>>>
>>> I am able to use ext4 partition read/write, created a file and wrote
>>> in it, un/re mounted it, all is ok.
>>> The drive only has a microusb port and a little led light; no rw
>>> switch or anything like that..
>>>
>>
>> What might help is running btrfs convert under strace. So something like :
>>
>> sudo strace -f -o strace.log btrfs-convert /dev/sdb1 and then send the log.
>>
> strace would greatly help in this case.
> 
> My guess is something wrong happened in migrate_super_block(), which
> doesn't handle ret > 0 case well.
> In that case, it means pwrite 4K doesn't finish in one call, which looks
> pretty strange.

So Qu,

I'm not seeing any EPERM errors, though I see the following so you might
be right:

openat(AT_FDCWD, "/dev/sdb1", O_RDWR) = 5

followed by a lot of preads, the last one of which is:

pread64(5, 0x557a0abd10b0, 4096, 2732765184) = -1 EIO (Input/output
error)

This is then followed by a a lot of pwrites, the last 10 or so syscalls
are:

pwrite64(5, "\220\255\262\244\0\0\0\0\0\0"..., 16384, 91127808) = 16384

pwrite64(5, "|IO\233\0\0\0\0\0\0"..., 16384, 91144192) = 16384

pwrite64(5, "x\2501n\0\0\0\0\0\0"..., 16384, 91160576) = 16384

pwrite64(5, "\252\254l)\0\0\0\0\0\0"..., 16384, 91176960) = 16384

pwrite64(5, "P\256\331\373\0\0\0\0\0\0"..., 16384, 91193344) = 16384

pwrite64(5, "\3\230\230+\0\0\0\0\0\0"..., 4096, 83951616) = 4096

write(2, "ERROR: ", 7) = 7

write(2, "failed to "..., 37) = 37

write(2, "\n", 1) = 1

close(4) = 0

write(2, "WARNING: ", 9) = 9

write(2, "an error o"..., 104) = 104

write(2, "\n", 1) = 1

exit_group(1) = ?

but looking at migrate_super_block it's not doing that many writes -
just writing the sb at BTRFS_SUPER_INFO_OFFSET and then zeroing out
0..BTRFS_SUPER_INFO_OFFSET. Also the number of pwrites:
grep -c pwrite ~/Downloads/btrfs-issue/convert-strace.log
198

So the failure is not originating from migrate_super_block.

> 
> Thanks,
> Qu
> 

Re: btrfs convert problem

[Qu Wenruo]

On 2018/9/14 下午1:52, Nikolay Borisov wrote:
> 
> 
> On 14.09.2018 02:17, Qu Wenruo wrote:
>>
>>
>> On 2018/9/14 上午12:37, Nikolay Borisov wrote:
>>>
>>>
>>> On 13.09.2018 19:15, Serhat Sevki Dincer wrote:
>>>>> -1 seems to be EPERM, is your device write-protected, readonly or
>>>>> something like that ?
>>>>
>>>> I am able to use ext4 partition read/write, created a file and wrote
>>>> in it, un/re mounted it, all is ok.
>>>> The drive only has a microusb port and a little led light; no rw
>>>> switch or anything like that..
>>>>
>>>
>>> What might help is running btrfs convert under strace. So something like :
>>>
>>> sudo strace -f -o strace.log btrfs-convert /dev/sdb1 and then send the log.
>>>
>> strace would greatly help in this case.
>>
>> My guess is something wrong happened in migrate_super_block(), which
>> doesn't handle ret > 0 case well.
>> In that case, it means pwrite 4K doesn't finish in one call, which looks
>> pretty strange.
> 
> So Qu,
> 
> I'm not seeing any EPERM errors, though I see the following so you might
> be right:
> 
> openat(AT_FDCWD, "/dev/sdb1", O_RDWR) = 5
> 
> followed by a lot of preads, the last one of which is:
> 
> pread64(5, 0x557a0abd10b0, 4096, 2732765184) = -1 EIO (Input/output
> error)

Would you please provide the full log?

This pread() looks like some data read.

But without context, it's a little hard to say.

Thanks,
Qu

> 
> This is then followed by a a lot of pwrites, the last 10 or so syscalls
> are:
> 
> pwrite64(5, "\220\255\262\244\0\0\0\0\0\0"..., 16384, 91127808) = 16384
> 
> pwrite64(5, "|IO\233\0\0\0\0\0\0"..., 16384, 91144192) = 16384
> 
> pwrite64(5, "x\2501n\0\0\0\0\0\0"..., 16384, 91160576) = 16384
> 
> pwrite64(5, "\252\254l)\0\0\0\0\0\0"..., 16384, 91176960) = 16384
> 
> pwrite64(5, "P\256\331\373\0\0\0\0\0\0"..., 16384, 91193344) = 16384
> 
> pwrite64(5, "\3\230\230+\0\0\0\0\0\0"..., 4096, 83951616) = 4096
> 
> write(2, "ERROR: ", 7) = 7
> 
> write(2, "failed to "..., 37) = 37
> 
> write(2, "\n", 1) = 1
> 
> close(4) = 0
> 
> write(2, "WARNING: ", 9) = 9
> 
> write(2, "an error o"..., 104) = 104
> 
> write(2, "\n", 1) = 1
> 
> exit_group(1) = ?
> 
> but looking at migrate_super_block it's not doing that many writes -
> just writing the sb at BTRFS_SUPER_INFO_OFFSET and then zeroing out
> 0..BTRFS_SUPER_INFO_OFFSET. Also the number of pwrites:
> grep -c pwrite ~/Downloads/btrfs-issue/convert-strace.log
> 198
> 
> So the failure is not originating from migrate_super_block.
> 
>>
>> Thanks,
>> Qu
>>

Re: btrfs convert problem

[Qu Wenruo]

On 2018/9/14 下午1:52, Nikolay Borisov wrote:
> 
> 
> On 14.09.2018 02:17, Qu Wenruo wrote:
>>
>>
>> On 2018/9/14 上午12:37, Nikolay Borisov wrote:
>>>
>>>
>>> On 13.09.2018 19:15, Serhat Sevki Dincer wrote:
>>>>> -1 seems to be EPERM, is your device write-protected, readonly or
>>>>> something like that ?
>>>>
>>>> I am able to use ext4 partition read/write, created a file and wrote
>>>> in it, un/re mounted it, all is ok.
>>>> The drive only has a microusb port and a little led light; no rw
>>>> switch or anything like that..
>>>>
>>>
>>> What might help is running btrfs convert under strace. So something like :
>>>
>>> sudo strace -f -o strace.log btrfs-convert /dev/sdb1 and then send the log.
>>>
>> strace would greatly help in this case.
>>
>> My guess is something wrong happened in migrate_super_block(), which
>> doesn't handle ret > 0 case well.
>> In that case, it means pwrite 4K doesn't finish in one call, which looks
>> pretty strange.
> 
> So Qu,
> 
> I'm not seeing any EPERM errors, though I see the following so you might
> be right:
> 
> openat(AT_FDCWD, "/dev/sdb1", O_RDWR) = 5
> 
> followed by a lot of preads, the last one of which is:
> 
> pread64(5, 0x557a0abd10b0, 4096, 2732765184) = -1 EIO (Input/output
> error)

With context, it's pretty easy to locate the problem.

It's a bad block of your device.

Please try to use the following command to verify it:

dd if=/dev/sdb1 of=/dev/null bs=1 count=4096 skip=2732765184

And it would be nicer to check kernel dmesg to see if there is clue there.

The culprit code is read_disk_extent(), which reset ret to -1 when error
happens.
I'll send patch(es) to fix it and enhance error messages for btrfs-convert.


There is a workaround to let btrfs-convert continue, you could use
--no-datasum option to skip datasum generation, thus btrfs-convert won't
try to read such data.
But that's just ostrich algorithm.

I'd recommend to read all files in the original fs and locate which
file(s) are affected. And backup data asap.

Thanks,
Qu

> 
> This is then followed by a a lot of pwrites, the last 10 or so syscalls
> are:
> 
> pwrite64(5, "\220\255\262\244\0\0\0\0\0\0"..., 16384, 91127808) = 16384
> 
> pwrite64(5, "|IO\233\0\0\0\0\0\0"..., 16384, 91144192) = 16384
> 
> pwrite64(5, "x\2501n\0\0\0\0\0\0"..., 16384, 91160576) = 16384
> 
> pwrite64(5, "\252\254l)\0\0\0\0\0\0"..., 16384, 91176960) = 16384
> 
> pwrite64(5, "P\256\331\373\0\0\0\0\0\0"..., 16384, 91193344) = 16384
> 
> pwrite64(5, "\3\230\230+\0\0\0\0\0\0"..., 4096, 83951616) = 4096
> 
> write(2, "ERROR: ", 7) = 7
> 
> write(2, "failed to "..., 37) = 37
> 
> write(2, "\n", 1) = 1
> 
> close(4) = 0
> 
> write(2, "WARNING: ", 9) = 9
> 
> write(2, "an error o"..., 104) = 104
> 
> write(2, "\n", 1) = 1
> 
> exit_group(1) = ?
> 
> but looking at migrate_super_block it's not doing that many writes -
> just writing the sb at BTRFS_SUPER_INFO_OFFSET and then zeroing out
> 0..BTRFS_SUPER_INFO_OFFSET. Also the number of pwrites:
> grep -c pwrite ~/Downloads/btrfs-issue/convert-strace.log
> 198
> 
> So the failure is not originating from migrate_super_block.
> 
>>
>> Thanks,
>> Qu
>>

Re: btrfs convert problem

[Serhat Sevki Dincer]

Hi,

On my second attempt to convert my 698 GiB usb HDD from ext4 to btrfs
with btrfs-progs 4.19 from Manjaro (kernel 4.14.80):

I identified bad files with
$ find . -type f -exec cat {} > /dev/null \;
This revealed 6 corrupted files, I deleted them. Tried it again with
no error message.

Then I checked HDD with
$ sudo fsck.ext4 -f /dev/sdb1
e2fsck 1.44.4 (18-Aug-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
HDD: 763513/45744128 files (0.3% non-contiguous), 131674747/182970301 blocks
It is mountable & usable.

Now my attempt to convert
$ LC_ALL=en_US.utf8 sudo strace -f -s 10 -a 4 -o convert-strace.log
btrfs-convert /dev/sdb1
create btrfs filesystem:
    blocksize: 4096
    nodesize:  16384
    features:  extref, skinny-metadata (default)
creating ext2 image file
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
ctree.c:2244: split_leaf: BUG_ON `1` triggered, value 1
btrfs-convert(+0x162d6)[0x561fbd26b2d6]
btrfs-convert(btrfs_search_slot+0xf21)[0x561fbd26c881]
btrfs-convert(btrfs_csum_file_block+0x499)[0x561fbd27e8e9]
btrfs-convert(+0xe6f5)[0x561fbd2636f5]
btrfs-convert(main+0x194f)[0x561fbd26296f]
/usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7f658a93a223]
btrfs-convert(_start+0x2e)[0x561fbd26328e]
Aborted

It crashed :) The log is 2.4G, I compressed it and put it at
https://drive.google.com/drive/folders/0B5oVWFBM47D9aVpJY2s4UTdMdUU

What else can I do to debug further?
Thanks..

Re: btrfs convert problem

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2262 bytes --]



On 2018/11/17 下午5:49, Serhat Sevki Dincer wrote:
> Hi,
> 
> On my second attempt to convert my 698 GiB usb HDD from ext4 to btrfs
> with btrfs-progs 4.19 from Manjaro (kernel 4.14.80):
> 
> I identified bad files with
> $ find . -type f -exec cat {} > /dev/null \;
> This revealed 6 corrupted files, I deleted them. Tried it again with
> no error message.
> 
> Then I checked HDD with
> $ sudo fsck.ext4 -f /dev/sdb1
> e2fsck 1.44.4 (18-Aug-2018)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> HDD: 763513/45744128 files (0.3% non-contiguous), 131674747/182970301 blocks
> It is mountable & usable.
> 
> Now my attempt to convert
> $ LC_ALL=en_US.utf8 sudo strace -f -s 10 -a 4 -o convert-strace.log
> btrfs-convert /dev/sdb1
> create btrfs filesystem:
>     blocksize: 4096
>     nodesize:  16384
>     features:  extref, skinny-metadata (default)
> creating ext2 image file
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0

It's ENOSPC.
Since it won't damage your original ext* fs, you could try to check if
there are enough *continuous* space for btrfs to use.

Please keep in mind that, due to the nature ext* data layout, it may
contain a lot of small free space fragments, and in that case
btrfs-convert may not be able to take use of them.

Or you could try to disable csum to make btrfs take less space so it may
have a chance to convert.

Thanks,
Qu
> ctree.c:2244: split_leaf: BUG_ON `1` triggered, value 1
> btrfs-convert(+0x162d6)[0x561fbd26b2d6]
> btrfs-convert(btrfs_search_slot+0xf21)[0x561fbd26c881]
> btrfs-convert(btrfs_csum_file_block+0x499)[0x561fbd27e8e9]
> btrfs-convert(+0xe6f5)[0x561fbd2636f5]
> btrfs-convert(main+0x194f)[0x561fbd26296f]
> /usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7f658a93a223]
> btrfs-convert(_start+0x2e)[0x561fbd26328e]
> Aborted
> 
> It crashed :) The log is 2.4G, I compressed it and put it at
> https://drive.google.com/drive/folders/0B5oVWFBM47D9aVpJY2s4UTdMdUU
> 
> What else can I do to debug further?
> Thanks..
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]