btrfs balancing start - and stop?

btrfs balancing start - and stop?

[Struan Bartlett]

Hi,

My company is testing btrfs (kernel 2.6.38) on a slave MySQL database 
server with a 195Gb filesystem (of which about 123Gb is used). So far, 
we're quite impressed with the performance. Our database loads are high, 
and if  filesystem performance wasn't good, MySQL replication wouldn't 
be able to keep up and the slave latency would begin to climb. This 
though, is generally not happening, which is good.

However, we recently tried running 'btrfs fi balance' on the filesystem, 
and found this deteriorated performance significantly, and the MySQL 
replication latency did begin to climb. Several hours later, with the 
btrfs-cleaner thread apparently still busy, and our replication latency 
running to a couple of hours, and no sign of the balancing operation 
finishing, we decided we needed to terminate the balancing operation, 
which we did by rebooting the server.

That, however, is suboptimal in a production environment, and so I've 
some questions.

1) Is the balancing operation expected to take many hours (or days?) on 
a filesystem such as this? Or are there known issues with the algorithm 
that are yet to be addressed?

2) Is it supposed to be desirable to run balancing operations 
periodically anyway? Our server is running on hardware mirrored disks, 
so our btrfs filesystem is simply created in spare space on the LVM 
volume group, using a single LV block device. Does balancing help 
improve performance/optimise free space in this setup anyway?

3) If there's an ioctl for launching a balancing operation, would it be 
an idea to add one for pausing a balancing operation? If balancing may 
take 'significant' lengths of time, and if it's intended that balancing 
be done periodically, it might be helpful if one could start balancing 
when loads are lower, and make sure one can stop them when resources are 
needed (in our case, when slave latency exceeds acceptable limits).

Kind regards

Struan Bartlett
CTO
NewsNow.co.uk

Re: btrfs balancing start - and stop?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3221 bytes --]

On Fri, Apr 01, 2011 at 12:14:50PM +0100, Struan Bartlett wrote:
> My company is testing btrfs (kernel 2.6.38) on a slave MySQL
> database server with a 195Gb filesystem (of which about 123Gb is
> used). So far, we're quite impressed with the performance. Our
> database loads are high, and if  filesystem performance wasn't good,
> MySQL replication wouldn't be able to keep up and the slave latency
> would begin to climb. This though, is generally not happening, which
> is good.
> 
> However, we recently tried running 'btrfs fi balance' on the
> filesystem, and found this deteriorated performance significantly,
> and the MySQL replication latency did begin to climb. Several hours
> later, with the btrfs-cleaner thread apparently still busy, and our
> replication latency running to a couple of hours, and no sign of the
> balancing operation finishing, we decided we needed to terminate the
> balancing operation, which we did by rebooting the server.
> 
> That, however, is suboptimal in a production environment, and so
> I've some questions.
> 
> 1) Is the balancing operation expected to take many hours (or days?)
> on a filesystem such as this? Or are there known issues with the
> algorithm that are yet to be addressed?

   A balance rewrites all the data on the filesystem, so it can take a
very long time (I think the longest reported time I've seen from
anyone was 48 hours, on several terabytes of data). However, this will
be highly dependent on the amount of I/O bandwidth available to the
FS, and on the size of the data to be written.

> 2) Is it supposed to be desirable to run balancing operations
> periodically anyway? Our server is running on hardware mirrored
> disks, so our btrfs filesystem is simply created in spare space on
> the LVM volume group, using a single LV block device. Does balancing
> help improve performance/optimise free space in this setup anyway?

   Not that I'm aware of, particularly in the light of the recent
patch that frees up unused block groups. Others here may have a more
informed take on this, though.

> 3) If there's an ioctl for launching a balancing operation, would it
> be an idea to add one for pausing a balancing operation? If
> balancing may take 'significant' lengths of time, and if it's
> intended that balancing be done periodically, it might be helpful if
> one could start balancing when loads are lower, and make sure one
> can stop them when resources are needed (in our case, when slave
> latency exceeds acceptable limits).

   There's patches for a cancel operation on the mailing list.
Further, I've got (as yet) unreleased patches for various forms of
partial balance, at least one of which would allow a balance to be
restarted after it was cancelled. The only reason I've not released
them is because I want to do a final check of what I send to the list
to ensure that I'm not making an idiot of myself (and wasting people's
time) with malformed patches. I hope to have time for this on Sunday.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- Guards!  Help! We're being rescued! ---               

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: btrfs balancing start - and stop?

[Struan Bartlett]

On 01/04/11 12:59, Hugo Mills wrote:
> On Fri, Apr 01, 2011 at 12:14:50PM +0100, Struan Bartlett wrote:
>    
>> My company is testing btrfs (kernel 2.6.38) on a slave MySQL
>> database server with a 195Gb filesystem (of which about 123Gb is
>> used). So far, we're quite impressed with the performance. Our
>> database loads are high, and if  filesystem performance wasn't good,
>> MySQL replication wouldn't be able to keep up and the slave latency
>> would begin to climb. This though, is generally not happening, which
>> is good.
>>
>> However, we recently tried running 'btrfs fi balance' on the
>> filesystem, and found this deteriorated performance significantly,
>> and the MySQL replication latency did begin to climb. Several hours
>> later, with the btrfs-cleaner thread apparently still busy, and our
>> replication latency running to a couple of hours, and no sign of the
>> balancing operation finishing, we decided we needed to terminate the
>> balancing operation, which we did by rebooting the server.
>>
>> That, however, is suboptimal in a production environment, and so
>> I've some questions.
>>
>> 1) Is the balancing operation expected to take many hours (or days?)
>> on a filesystem such as this? Or are there known issues with the
>> algorithm that are yet to be addressed?
>>      
>     A balance rewrites all the data on the filesystem, so it can take a
> very long time (I think the longest reported time I've seen from
> anyone was 48 hours, on several terabytes of data). However, this will
> be highly dependent on the amount of I/O bandwidth available to the
> FS, and on the size of the data to be written.
>
>    
>> 2) Is it supposed to be desirable to run balancing operations
>> periodically anyway? Our server is running on hardware mirrored
>> disks, so our btrfs filesystem is simply created in spare space on
>> the LVM volume group, using a single LV block device. Does balancing
>> help improve performance/optimise free space in this setup anyway?
>>      
>     Not that I'm aware of, particularly in the light of the recent
> patch that frees up unused block groups. Others here may have a more
> informed take on this, though.
>
>    
>> 3) If there's an ioctl for launching a balancing operation, would it
>> be an idea to add one for pausing a balancing operation? If
>> balancing may take 'significant' lengths of time, and if it's
>> intended that balancing be done periodically, it might be helpful if
>> one could start balancing when loads are lower, and make sure one
>> can stop them when resources are needed (in our case, when slave
>> latency exceeds acceptable limits).
>>      
>     There's patches for a cancel operation on the mailing list.
> Further, I've got (as yet) unreleased patches for various forms of
> partial balance, at least one of which would allow a balance to be
> restarted after it was cancelled. The only reason I've not released
> them is because I want to do a final check of what I send to the list
> to ensure that I'm not making an idiot of myself (and wasting people's
> time) with malformed patches. I hope to have time for this on Sunday.
>
>     Hugo.
>
>    
Hugo - thanks very much for your thorough reply. I look forward to being 
able to cancel a balancing operation, but in the meantime we simply 
won't bother setting any going, and see how things go. So far, our btrfs 
slave database has been running two weeks, with a rolling history of 
snapshots taken every ten minutes, without any other apparent issues.

Struan

Re: btrfs balancing start - and stop?

[Helmut Hullen]

Hallo, Struan,

Du meintest am 01.04.11:

> 1) Is the balancing operation expected to take many hours (or days?)
> on a filesystem such as this? Or are there known issues with the
> algorithm that are yet to be addressed?

May be. Balancing about 15 GByte needed about 2 hours (or less),  
balancing about 2 TByte needed about 20 hours.

"dmesg" counts down the number of remaining "jobs".

Viele Gruesse!
Helmut

Re: btrfs balancing start - and stop?

[Konstantinos Skarlatos]

On 1/4/2011 3:12 =CE=BC=CE=BC, Helmut Hullen wrote:
> Hallo, Struan,
>
> Du meintest am 01.04.11:
>
>> 1) Is the balancing operation expected to take many hours (or days?)
>> on a filesystem such as this? Or are there known issues with the
>> algorithm that are yet to be addressed?
> May be. Balancing about 15 GByte needed about 2 hours (or less),
> balancing about 2 TByte needed about 20 hours.
>
> "dmesg" counts down the number of remaining "jobs".
are you sure? here is a snippet of dmesg from a balance i did yesterday=
=20
(2.6.38.1)

btrfs: relocating block group 15338569728 flags 9
btrfs: found 17296 extents
btrfs: found 17296 extents
btrfs: relocating block group 13191086080 flags 9
btrfs: found 21029 extents
btrfs: found 21029 extents
btrfs: relocating block group 11043602432 flags 9
btrfs: found 4728 extents
btrfs: found 4728 extents


> Viele Gruesse!
> Helmut
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs balancing start - and stop?

[Helmut Hullen]

Hallo, Konstantinos,

Du meintest am 01.04.11:

>> "dmesg" counts down the number of remaining "jobs".

> are you sure? here is a snippet of dmesg from a balance i did
> yesterday (2.6.38.1)

> btrfs: relocating block group 15338569728 flags 9
> btrfs: found 17296 extents
> btrfs: found 17296 extents
> btrfs: relocating block group 13191086080 flags 9
> btrfs: found 21029 extents
> btrfs: found 21029 extents
> btrfs: relocating block group 11043602432 flags 9
> btrfs: found 4728 extents
> btrfs: found 4728 extents

Yes - there I look how long the balancing job may still work. You see  
that the "relocating" line counts down: 15 13 11 ...

Viele Gruesse!
Helmut

Re: btrfs balancing start - and stop?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1740 bytes --]

On Fri, Apr 01, 2011 at 03:36:00PM +0200, Helmut Hullen wrote:
> Hallo, Konstantinos,
> 
> Du meintest am 01.04.11:
> 
> >> "dmesg" counts down the number of remaining "jobs".
> 
> > are you sure? here is a snippet of dmesg from a balance i did
> > yesterday (2.6.38.1)
> 
> > btrfs: relocating block group 15338569728 flags 9
> > btrfs: found 17296 extents
> > btrfs: found 17296 extents
> > btrfs: relocating block group 13191086080 flags 9
> > btrfs: found 21029 extents
> > btrfs: found 21029 extents
> > btrfs: relocating block group 11043602432 flags 9
> > btrfs: found 4728 extents
> > btrfs: found 4728 extents
> 
> Yes - there I look how long the balancing job may still work. You see  
> that the "relocating" line counts down: 15 13 11 ...

   It's not a good measure of time remaining, as the numbers there are
pretty arbitrary: they're the start of the block group in btrfs's own
internal address space. The balance algorithm will indeed go through
the block groups in reverse order, but they're not guaranteed to
terminate at zero. In fact, if you've balanced before, they're pretty
much guaranteed to terminate a long way before zero.

   New block groups (for example, ones created during a balance) are
always created with virtual addresses larger than any previous block
group. So, if you started with block groups at addresses 1G, 2G, 3G,
4G and balanced, you'd end up with ones at 5G, 6G, 7G, 8G. The next
time you balanced, you'd see the numbers count down: 8G, 7G, 6G, 5G,
complete.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
             --- Happiness is mandatory.  Are you happy? ---             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: btrfs balancing start - and stop?

[Konstantinos Skarlatos]

On 1/4/2011 4:37 =CE=BC=CE=BC, Hugo Mills wrote:
> On Fri, Apr 01, 2011 at 04:22:39PM +0300, Konstantinos Skarlatos wrot=
e:
>> On 1/4/2011 3:12 =CE=BC=CE=BC, Helmut Hullen wrote:
>>> Du meintest am 01.04.11:
>>>
>>> "dmesg" counts down the number of remaining "jobs".
>> are you sure? here is a snippet of dmesg from a balance i did
>> yesterday (2.6.38.1)
>>
>> btrfs: relocating block group 15338569728 flags 9
>> btrfs: found 17296 extents
>> btrfs: found 17296 extents
>> btrfs: relocating block group 13191086080 flags 9
>> btrfs: found 21029 extents
>> btrfs: found 21029 extents
>> btrfs: relocating block group 11043602432 flags 9
>> btrfs: found 4728 extents
>> btrfs: found 4728 extents
>     Count the number of block groups in the system (1GiB for data,
> 256MiB for metdata on a typical filesystem), and subtract the number
> of "relocating block group" messages... Not ideal, but it's possible.
>
>     The balance cancel patch I mentioned earlier also supplies an
> additional patch for monitoring progress, which does show up in the
> dmesg output (as well as user-space support for prettier output).
Great, I think it is very important to have a human-readable progress=20
monitor for operations like that.
>     Hugo.
>

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs balancing start - and stop?

[Stephane Chazelas]

On Fri, 2011-04-01 at 14:12 +0200, Helmut Hullen wrote:
> Hallo, Struan,
> 
> Du meintest am 01.04.11:
> 
> > 1) Is the balancing operation expected to take many hours (or days?)
> > on a filesystem such as this? Or are there known issues with the
> > algorithm that are yet to be addressed?
> 
> May be. Balancing about 15 GByte needed about 2 hours (or less),  
> balancing about 2 TByte needed about 20 hours.
[...]

I've got a balance running since Monday on a 9TB volume (3.5 of which
are used, 3.2 allegedly free), showing no sign of finishing soon. Should
I be worried?

Using /proc/sys/vm/block_dump, I can see it's seeking all over the
place, which is probably why throughput is not high. I can also see it
writing several times to the same sectors.

# df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             8.2T  3.5T  3.2T  53% /backup
# btrfs fi sh
Label: none  uuid: ...
        Total devices 3 FS bytes used 3.43TB
        devid    4 size 2.73TB used 1.16TB path /dev/sdc
        devid    3 size 2.73TB used 1.16TB path /dev/sdb
        devid    2 size 2.70TB used 1.14TB path /dev/sda4

Btrfs Btrfs v0.19
# ps -eolstart,args | grep balance
Mon Mar 28 11:18:18 2011 sudo btrfs fi balance /backup
Mon Mar 28 11:18:18 2011 btrfs fi balance /backup
# date
Fri Apr  1 19:28:40 BST 2011
# btrfs fi df /backup
Data, RAID0: total=3.41TB, used=3.41TB
System, RAID1: total=16.00MB, used=232.00KB
Metadata, RAID1: total=27.75GB, used=20.47GB
# iostat -md
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              14.49         2.37         2.39     903123     913112
sdc             501.23         2.68         5.06    1022456    1928462
sdb             477.28         2.58         5.06     982853    1928482

It's already written more than the used space.

Cheers,
Stephane

Re: btrfs balancing start - and stop?

[Helmut Hullen]

Hallo, Stephane,

Du meintest am 01.04.11:

>> balancing about 2 TByte needed about 20 hours.
> [...]

> I've got a balance running since Monday on a 9TB volume (3.5 of which
> are used, 3.2 allegedly free), showing no sign of finishing soon.
> Should I be worried?

> Using /proc/sys/vm/block_dump, I can see it's seeking all over the
> place, which is probably why throughput is not high. I can also see
> it writing several times to the same sectors.


Hugo has explained the limits of regarding

        dmesg | grep relocating

or (more simple) the last lines of "dmesg" and looking for the  
"relocating" lines. But: what do these lines tell now? What is the  
(pessimistic) estimation when you extrapolate the data?

(please excuse my gerlish)

Viele Gruesse!
Helmut

Re: btrfs balancing start - and stop?

[Stephane Chazelas]

2011-04-01 21:26:00 +0200, Helmut Hullen:
> Hallo, Stephane,

Hi Helmut,

> Du meintest am 01.04.11:
> 
> >> balancing about 2 TByte needed about 20 hours.
> > [...]
> 
> > I've got a balance running since Monday on a 9TB volume (3.5 of which
> > are used, 3.2 allegedly free), showing no sign of finishing soon.
> > Should I be worried?
> 
> > Using /proc/sys/vm/block_dump, I can see it's seeking all over the
> > place, which is probably why throughput is not high. I can also see
> > it writing several times to the same sectors.
> 
> 
> Hugo has explained the limits of regarding
> 
>         dmesg | grep relocating
> 
> or (more simple) the last lines of "dmesg" and looking for the  
> "relocating" lines. But: what do these lines tell now? What is the  
> (pessimistic) estimation when you extrapolate the data?
> 
> (please excuse my gerlish)
[...]

$ dmesg | grep reloc | sed -e 1b -e '$!d'
[370178.209571] btrfs: relocating block group 5612075220992 flags 20
[546163.062739] btrfs: relocating block group 3923616202752 flags 20

So, if it's to go down to zero:

$ echo $((3923616202752*(546163.062739-370178.209571)/(5612075220992-3923616202752)/86400))
4.7332292856705722

4.7 more days to go. And I reckon it will have written about 9
TB to disk by that time (which is the total size of the volume,
though only 3.8TB are occupied).

-- 
Stephane

Re: btrfs balancing start - and stop?

[Helmut Hullen]

Hallo, Stephane,

Du meintest am 03.04.11:

>>>> balancing about 2 TByte needed about 20 hours.

[...]

>> Hugo has explained the limits of regarding
>>
>>         dmesg | grep relocating
>>
>> or (more simple) the last lines of "dmesg" and looking for the
>> "relocating" lines. But: what do these lines tell now? What is the
>> (pessimistic) estimation when you extrapolate the data?

[...]

> 4.7 more days to go. And I reckon it will have written about 9
> TB to disk by that time (which is the total size of the volume,
> though only 3.8TB are occupied).

Yes - that's the pessimistic estimation. As Hugo has explained it can  
finish faster - just look to the data tomorrow again.

Viele Gruesse!
Helmut

Re: btrfs balancing start - and stop?

[Stephane Chazelas]

2011-04-03 21:35:00 +0200, Helmut Hullen:
> Hallo, Stephane,
> 
> Du meintest am 03.04.11:
> 
> >>>> balancing about 2 TByte needed about 20 hours.
> 
> [...]
> 
> >> Hugo has explained the limits of regarding
> >>
> >>         dmesg | grep relocating
> >>
> >> or (more simple) the last lines of "dmesg" and looking for the
> >> "relocating" lines. But: what do these lines tell now? What is the
> >> (pessimistic) estimation when you extrapolate the data?
> 
> [...]
> 
> > 4.7 more days to go. And I reckon it will have written about 9
> > TB to disk by that time (which is the total size of the volume,
> > though only 3.8TB are occupied).
> 
> Yes - that's the pessimistic estimation. As Hugo has explained it can  
> finish faster - just look to the data tomorrow again.
[...]

That may be an optimistic estimation actually, as there hasn't
been much progress in the last 34 hours:

# dmesg | awk -F '[][ ]+' '/reloc/ &&n++%5==0 {x=(n-$7)/($2-t)/1048576; printf "%s\t%s\t%.2f\t%*s\n", $2/3600,$7, x, x/3, ""; t=$2; n=$7}' | tr ' ' '*' | tail -40
125.629 4170039951360   11.93   ***
125.641 4166818725888   70.99   ***********************
125.699 4157155049472   43.87   **************
125.753 4144270147584   63.34   *********************
125.773 4137827696640   84.98   ****************************
125.786 4134606471168   64.39   *********************
125.823 4124942794752   70.09   ***********************
125.87  4112057892864   71.66   ***********************
125.887 4105615441920   100.60  *********************************
125.898 4102394216448   81.26   ***************************
125.935 4092730540032   69.06   ***********************
126.33  4085751218176   4.69    *
131.904 4072597880832   0.63
132.082 4059712978944   19.20   ******
132.12  4053270528000   45.52   ***************
132.138 4050049302528   45.60   ***************
132.225 4040385626112   29.68   *********
132.267 4027500724224   81.17   ***************************
132.283 4021058273280   106.31  ***********************************
132.29  4017837047808   110.42  ************************************
132.316 4008173371392   100.54  *********************************
132.358 3995288469504   81.18   ***************************
132.475 3988846018560   14.62   ****
132.514 3985624793088   21.55   *******
132.611 3975961116672   26.40   ********
132.663 3963076214784   65.31   *********************
132.678 3956633763840   120.11  ****************************************
132.685 3956365328384   10.26   ***
137.701 3949922877440   0.34
137.709 3946701651968   106.54  ***********************************
137.744 3937037975552   72.10   ************************
137.889 3927105863680   18.18   ******
137.901 3926837428224   5.85    *
141.555 3926300557312   0.04
141.93  3925226815488   0.76
151.227 3924421509120   0.02
151.491 3924153073664   0.27
151.712 3923616202752   0.64
165.301 3922542460928   0.02
174.346 3921737154560   0.02

At this rate (third field expressed in MiB/s), it could take
months to complete.

iostat still reports writes at about 5MiB/s though. Note that
this system is not doing anything else at all.

There definitely seems to be scope for optimisation in the
"balancing" I'd say.

-- 
Stephane

Re: btrfs balancing start - and stop?

[Stephane Chazelas]

2011-04-04 20:07:54 +0100, Stephane Chazelas:
[...]
> > > 4.7 more days to go. And I reckon it will have written about 9
> > > TB to disk by that time (which is the total size of the volume,
> > > though only 3.8TB are occupied).
> > 
> > Yes - that's the pessimistic estimation. As Hugo has explained it can  
> > finish faster - just look to the data tomorrow again.
> [...]
> 
> That may be an optimistic estimation actually, as there hasn't
> been much progress in the last 34 hours:
[...]

The rate is going down. It's now down to about 14kB/s

[658654.295752] btrfs: relocating block group 3919858106368 flags 20
[671932.913235] btrfs: relocating block group 3919589670912 flags 20
[686189.296126] btrfs: relocating block group 3919321235456 flags 20
[701511.523990] btrfs: relocating block group 3919052800000 flags 20
[718591.316339] btrfs: relocating block group 3918784364544 flags 20
[725567.081031] btrfs: relocating block group 3918515929088 flags 20
[744415.011581] btrfs: relocating block group 3918247493632 flags 20
[762365.021458] btrfs: relocating block group 3917979058176 flags 20
[780504.726067] btrfs: relocating block group 3917710622720 flags 20

Even though it is reading and writing to disk at a much higher
rate. Here stats every second:

--dsk/sda-----dsk/sdb-----dsk/sdc--
 read  writ: read  writ: read  writ
   0     0 : 540k    0 :  12k    0
   0     0 : 704k    0 :  20k    0
   0     0 :1068k    0 :  24k    0
   0     0 : 968k    0 :   0     0
   0     0 : 932k    0 :4096B    0
   0     0 : 832k  880k: 152k 1320k
  60k 4096B: 880k  140k:   0    28M
  68k    0 : 308k    0 :4096B 9240k
   0    48k:   0     0 :   0  7852k
   0     0 : 576k 6192k:4096B   26M
   0     0 : 100k   18M:   0     0
   0     0 :  28k   10M:   0     0
   0     0 :   0  7020k:   0     0
   0     0 :  52k   13M:   0     0
   0    12k: 528k   17M:   0    12k
   0     0 : 884k    0 :8192B    0
   0     0 :1068k    0 :  20k    0
   0     0 : 660k    0 :   0     0
   0    40k: 776k    0 :4096B    0
   0     0 : 576k    0 :   0     0
   0     0 : 596k    0 :8192B    0
1096k   28k: 664k    0 :4096B    0
   0     0 : 660k    0 :   0     0
   0     0 : 592k    0 :8192B    0

At this rate, the balancing would be over in about 8 years.

Since the start of the balance:
Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              10.04         1.56         1.57    1228286    1237359
sdc             396.24         1.77         3.95    1397015    3115057
sdb             421.17         1.87         3.95    1473759    3115093

I think that's the end of my attempt to transfer that FS to
another machine (see other thread). I'll have to ditch that copy
and try again from scratch with another approach.

Before I do that, is there anything I can do to help investigate
the problem?

regards,
Stephane

Re: btrfs balancing start - and stop?

[Stephane Chazelas]

2011-04-06 12:43:50 +0100, Stephane Chazelas:
[...]
> The rate is going down. It's now down to about 14kB/s
> 
> [658654.295752] btrfs: relocating block group 3919858106368 flags 20
> [671932.913235] btrfs: relocating block group 3919589670912 flags 20
> [686189.296126] btrfs: relocating block group 3919321235456 flags 20
> [701511.523990] btrfs: relocating block group 3919052800000 flags 20
> [718591.316339] btrfs: relocating block group 3918784364544 flags 20
> [725567.081031] btrfs: relocating block group 3918515929088 flags 20
> [744415.011581] btrfs: relocating block group 3918247493632 flags 20
> [762365.021458] btrfs: relocating block group 3917979058176 flags 20
> [780504.726067] btrfs: relocating block group 3917710622720 flags 20
[...]
> At this rate, the balancing would be over in about 8 years.
[...]

Hurray! The btrfs balance eventually ran through after almost exactly 2 weeks.
It didn't get down to 0:

[1189505.152717] btrfs: found 60527 extents
[1189505.440565] btrfs: relocating block group 3910731300864 flags 20
[1199805.071045] btrfs: found 60235 extents
[1199805.447821] btrfs: relocating block group 3910462865408 flags 20
[1207914.737372] btrfs: found 58039 extents

iostat reckons 9TB have been written to  disk in the whole
process (4.5TB read from them (!?)).

There hasn't been any change in allocation though:

# df -h /backup
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             8.2T  3.5T  3.2T  53% /backup
# btrfs fi df /backup
Data, RAID0: total=3.42TB, used=3.41TB
System, RAID1: total=16.00MB, used=228.00KB
Metadata, RAID1: total=28.00GB, used=20.47GB
# btrfs fi show
Label: none  uuid: a0ae35c4-51f2-405f-a4bb-e4f134b1d193
        Total devices 3 FS bytes used 3.43TB
        devid    4 size 2.73TB used 1.17TB path /dev/sdc
        devid    3 size 2.73TB used 1.17TB path /dev/sdb
        devid    2 size 2.70TB used 1.14TB path /dev/sda4

Btrfs Btrfs v0.19

Still 1.5TB missing.

-- 
Stephane

Re: btrfs balancing start - and stop?

[Helmut Hullen]

Hallo, Stephane,

Du meintest am 11.04.11:

>> [780504.726067] btrfs: relocating block group 3917710622720 flags 20
> [...]
>> At this rate, the balancing would be over in about 8 years.
> [...]

> Hurray! The btrfs balance eventually ran through after almost exactly
> 2 weeks. It didn't get down to 0:

Congratulations!

> There hasn't been any change in allocation though:

> # df -h /backup
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda4             8.2T  3.5T  3.2T  53% /backup

> # btrfs fi df /backup
> Data, RAID0: total=3.42TB, used=3.41TB
> System, RAID1: total=16.00MB, used=228.00KB
> Metadata, RAID1: total=28.00GB, used=20.47GB

> Still 1.5TB missing.

Seems to be the same problem I've just mourned about.

Just expand "available" to "at least available". And ignore the value in  
"Data, RAID0: total=3.42TB".

Viele Gruesse!
Helmut