btrfs-convert processing time

btrfs-convert processing time

[Olivier Bonvalet]

Hi,

I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the 
time needed to do that conversion.

The first FS is on a 500GiB block device, and btrfs-convert is running 
since more than 48h :
root      1978 25.6 47.7 748308 732556 ?       D    Feb18 944:44 
btrfs-convert /dev/vg-backup/backup

The second is on a 340GiB block device, and the processing time is similar :
root     30192  5.2  3.0  73180 61520 pts/0    R    Feb18 121:04 
btrfs-convert /dev/vg-balor/backup


Is it "normal", or is there a problem during the processing ?

Thanks,
Olivier

Re: btrfs-convert processing time

[Hubert Kario]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
> Hi,
> 
> I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the
> time needed to do that conversion.
> 
> The first FS is on a 500GiB block device, and btrfs-convert is running
> since more than 48h :
> root      1978 25.6 47.7 748308 732556 ?       D    Feb18 944:44
> btrfs-convert /dev/vg-backup/backup
> 
> The second is on a 340GiB block device, and the processing time is similar :
> root     30192  5.2  3.0  73180 61520 pts/0    R    Feb18 121:04
> btrfs-convert /dev/vg-balor/backup
> 
> 
> Is it "normal", or is there a problem during the processing ?
> 
> Thanks,
> Olivier

How much free space did those FS have? How many files were on them (was they 
dominated by small or large files)?

Regards,
-- 
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1794 bytes --]

Re: btrfs-convert processing time

[Hubert Kario]

(sorry for the duplicate, previous one has broken signature)
On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
> Hi,
>=20
> I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the
> time needed to do that conversion.
>=20
> The first FS is on a 500GiB block device, and btrfs-convert is runnin=
g
> since more than 48h :
> root      1978 25.6 47.7 748308 732556 ?       D    Feb18 944:44
> btrfs-convert /dev/vg-backup/backup
>=20
> The second is on a 340GiB block device, and the processing time is si=
milar :
> root     30192  5.2  3.0  73180 61520 pts/0    R    Feb18 121:04
> btrfs-convert /dev/vg-balor/backup
>=20
>=20
> Is it "normal", or is there a problem during the processing ?
>=20
> Thanks,
> Olivier

How much free space did those FS have? How many files were on them (was=
 they=20
dominated by small or large files)?

Regards,
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert processing time

[Olivier Bonvalet]

On 20/02/2012 14:20, Hubert Kario wrote:
> On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
>> Hi,
>>
>> I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the
>> time needed to do that conversion.
>>
>> The first FS is on a 500GiB block device, and btrfs-convert is running
>> since more than 48h :
>> root      1978 25.6 47.7 748308 732556 ?       D    Feb18 944:44
>> btrfs-convert /dev/vg-backup/backup
>>
>> The second is on a 340GiB block device, and the processing time is similar :
>> root     30192  5.2  3.0  73180 61520 pts/0    R    Feb18 121:04
>> btrfs-convert /dev/vg-balor/backup
>>
>>
>> Is it "normal", or is there a problem during the processing ?
>>
>> Thanks,
>> Olivier
>
> How much free space did those FS have? How many files were on them (was they
> dominated by small or large files)?
>
> Regards,


Lot of small files (like compressed email from Maildir), and lot of 
hardlinks, and probably low free space (near 15% I suppose).


So I think I have my answer :)

Thanks

Re: btrfs-convert processing time

[Hubert Kario]

On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
> On 20/02/2012 14:20, Hubert Kario wrote:
> > On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
> >> Hi,
> >>=20
> >> I'm trying to convert two ext4 FS to btrfs, but I'm surprised by t=
he
> >> time needed to do that conversion.
> >>=20
> >> The first FS is on a 500GiB block device, and btrfs-convert is run=
ning
> >> since more than 48h :
> >> root      1978 25.6 47.7 748308 732556 ?       D    Feb18 944:44
> >> btrfs-convert /dev/vg-backup/backup
> >>=20
> >> The second is on a 340GiB block device, and the processing time is
> >> similar : root     30192  5.2  3.0  73180 61520 pts/0    R    Feb1=
8
> >> 121:04 btrfs-convert /dev/vg-balor/backup
> >>=20
> >>=20
> >> Is it "normal", or is there a problem during the processing ?
> >>=20
> >> Thanks,
> >> Olivier
> >=20
> > How much free space did those FS have? How many files were on them =
(was
> > they dominated by small or large files)?
> >=20
> > Regards,
>=20
> Lot of small files (like compressed email from Maildir), and lot of
> hardlinks, and probably low free space (near 15% I suppose).
>=20
>=20
> So I think I have my answer :)
>=20

Yes, this is probably the worst possible combination.

Plese keep us updated. Just to have exact numbers for new users.

Regards,
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert processing time

[Fajar A. Nugraha]

On Mon, Feb 20, 2012 at 8:50 PM, Hubert Kario <hka@qbs.com.pl> wrote:
> On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
>> Lot of small files (like compressed email from Maildir), and lot of
>> hardlinks, and probably low free space (near 15% I suppose).
>>
>>
>> So I think I have my answer :)
>>
>
> Yes, this is probably the worst possible combination.
>
> Plese keep us updated. Just to have exact numbers for new users.


... although it would probably fail anyway due to btrfs hardlink limit
in the same directory.

-- 
Fajar

Re: btrfs-convert processing time

[Hubert Kario]

Chris: What will btrfs-convert do when it encounters a directory with m=
ore=20
hardlinks than the btrfs limit?

On Monday 20 of February 2012 21:00:34 Fajar A. Nugraha wrote:
> On Mon, Feb 20, 2012 at 8:50 PM, Hubert Kario <hka@qbs.com.pl> wrote:
> > On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
> >> Lot of small files (like compressed email from Maildir), and lot o=
f
> >> hardlinks, and probably low free space (near 15% I suppose).
> >>=20
> >>=20
> >> So I think I have my answer :)
> >=20
> > Yes, this is probably the worst possible combination.
> >=20
> > Plese keep us updated. Just to have exact numbers for new users.
>=20
> ... although it would probably fail anyway due to btrfs hardlink limi=
t
> in the same directory.

This problem also went through my mind.

Regards,
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert processing time

[Olivier Bonvalet]

On 20/02/2012 15:00, Fajar A. Nugraha wrote:
> On Mon, Feb 20, 2012 at 8:50 PM, Hubert Kario<hka@qbs.com.pl>  wrote:
>> On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
>>> Lot of small files (like compressed email from Maildir), and lot of
>>> hardlinks, and probably low free space (near 15% I suppose).
>>>
>>>
>>> So I think I have my answer :)
>>>
>>
>> Yes, this is probably the worst possible combination.
>>
>> Plese keep us updated. Just to have exact numbers for new users.
>
>
> ... although it would probably fail anyway due to btrfs hardlink limit
> in the same directory.
>

And in that case, btrfs-convert will abort, or ignore the error, or just 
hang ?

Re: btrfs-convert processing time

[Fajar A. Nugraha]

On Mon, Feb 20, 2012 at 9:29 PM, Olivier Bonvalet <btrfs.list@daevel.fr=
> wrote:
> On 20/02/2012 15:00, Fajar A. Nugraha wrote:
>>
>> On Mon, Feb 20, 2012 at 8:50 PM, Hubert Kario<hka@qbs.com.pl> =A0wro=
te:
>>>
>>> On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
>>>>
>>>> Lot of small files (like compressed email from Maildir), and lot o=
f
>>>> hardlinks, and probably low free space (near 15% I suppose).
>>>>
>>>>
>>>> So I think I have my answer :)
>>>>
>>>
>>> Yes, this is probably the worst possible combination.
>>>
>>> Plese keep us updated. Just to have exact numbers for new users.
>>
>>
>>
>> ... although it would probably fail anyway due to btrfs hardlink lim=
it
>> in the same directory.
>>
>
> And in that case, btrfs-convert will abort, or ignore the error, or j=
ust
> hang ?

On my simple test with ubuntu precise, loop-mounted ext4, 8k hardlinks:

$ sudo btrfs-convert /dev/loop0
creating btrfs metadata.
$ echo $?
139

so no useful error message, but it doesn't crash. And when mounted the
device still shows ext4.

A successful conversionn would look like this:

$ sudo btrfs-convert /dev/loop1
creating btrfs metadata.
creating ext2fs image file.
cleaning up system chunk.
conversion complete.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert processing time

[Olivier Bonvalet]

So, the btrfs-convert for the smaller drive is done... after near 5 
days. Which stats can I give you ?

It's a 340GB LVM block device, and "btrfs filesystem df /backup/" say that :

Data: total=225.97GB, used=181.94GB
System: total=32.00MB, used=24.00KB
Metadata: total=111.00GB, used=91.56GB


If I mount the ext2_saved/image copy, I can see 257GB of data (78% of 
the block device used), with 17M of inodes.


The other btrfs-convert stay running.

Re: btrfs-convert processing time

[Olivier Bonvalet]

Le 22/02/2012 17:55, Olivier Bonvalet a =E9crit :
> So, the btrfs-convert for the smaller drive is done... after near 5 d=
ays. Which stats can I give you ?
>
> It's a 340GB LVM block device, and "btrfs filesystem df /backup/" say=
 that :
>
> Data: total=3D225.97GB, used=3D181.94GB
> System: total=3D32.00MB, used=3D24.00KB
> Metadata: total=3D111.00GB, used=3D91.56GB
>
>
> If I mount the ext2_saved/image copy, I can see 257GB of data (78% of=
 the block device used), with 17M of inodes.
>
>
> The other btrfs-convert stay running.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

Well, the second one is still running :

root! backup:~# uptime
  21:17:43 up 41 days, 19:59,  1 user,  load average: 2.06, 1.90, 1.88


root! backup:~# ps auxw | grep btrfs
root      1978 25.6 74.3 1269072 1141212 ?     D    Feb18 15421:50 btrf=
s-convert /dev/vg-backup/backup


root! backup:~# iostat -k
Linux 2.6.42.3-dae-xen (backup) 	30/03/2012 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           10,54    0,06    2,36   54,21    0,09   32,74

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
xvda            724,70       175,50      3181,13  634342597 11498370400
xvdy              0,00         0,00         0,00       1381          0
xvdz              0,00         0,00         0,00       4065          0
dm-0              0,49         3,22         1,78   11627497    6435028
dm-1            837,57       171,69      3178,57  620593936 11489105416
dm-2              0,34         0,59         0,78    2120800    2830584

11498370400 kB written, so, 11TB written to convert a 518GB partition ?


but more important : is it safe to abort the process ?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-convert processing time

[Hubert Kario]

On Friday 30 of March 2012 21:23:52 Olivier Bonvalet wrote:
> Le 22/02/2012 17:55, Olivier Bonvalet a =E9crit :
> > So, the btrfs-convert for the smaller drive is done... after near 5
> > days. Which stats can I give you ?
> >=20
> > It's a 340GB LVM block device, and "btrfs filesystem df /backup/" s=
ay
> > that :
> >=20
> > Data: total=3D225.97GB, used=3D181.94GB
> > System: total=3D32.00MB, used=3D24.00KB
> > Metadata: total=3D111.00GB, used=3D91.56GB
> >=20
> >=20
> > If I mount the ext2_saved/image copy, I can see 257GB of data (78% =
of
> > the block device used), with 17M of inodes.
> >=20
> >=20
> > The other btrfs-convert stay running.
>=20
> Well, the second one is still running :
>=20
> root! backup:~# uptime
>   21:17:43 up 41 days, 19:59,  1 user,  load average: 2.06, 1.90, 1.8=
8
>=20
>=20
> root! backup:~# ps auxw | grep btrfs
> root      1978 25.6 74.3 1269072 1141212 ?     D    Feb18 15421:50
> btrfs-convert /dev/vg-backup/backup

Wow, that's /long/...


> root! backup:~# iostat -k
> Linux 2.6.42.3-dae-xen (backup) 	30/03/2012 	_x86_64_	(2 CPU)
>=20
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            10,54    0,06    2,36   54,21    0,09   32,74
>=20
> Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrt=
n
> xvda            724,70       175,50      3181,13  634342597 114983704=
00
> xvdy              0,00         0,00         0,00       1381          =
0
> xvdz              0,00         0,00         0,00       4065          =
0
> dm-0              0,49         3,22         1,78   11627497    643502=
8
> dm-1            837,57       171,69      3178,57  620593936 114891054=
16
> dm-2              0,34         0,59         0,78    2120800    283058=
4
>=20
> 11498370400 kB written, so, 11TB written to convert a 518GB partition=
 ?
>=20
>=20
> but more important : is it safe to abort the process ?

I'm quite sure you won't get a mountable filesystem, either ext or btrf=
s.=20
btrfs-convert has to move at least some ext3 metadata blocks, so ext FS=
 will=20
be broken. I don't know if btrfs-convert writes whole btrfs tree and th=
en=20
updates it as it finds new inodes in ext3 or just goes over it sequenti=
ally.=20
But either way, you'll get at best btrfs with lots of files missing.

So the short answer is: no.

Regards,
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html