Re: Stuck balance, 3.18.0

[Hugo Mills <hugo@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 2642 bytes --]

On Mon, Jan 12, 2015 at 01:36:27PM +0000, Holger Hoffstätte wrote:
> On Mon, 12 Jan 2015 12:27:12 +0000, Hugo Mills wrote:
> 
> > On Mon, Jan 12, 2015 at 11:21:58AM +0000, Hugo Mills wrote:
> >>    I've just added a new disk to my main storage filesystem. Running
> >> the obligatory balance to spread the data out, it's managed about 14%
> >> of the job, and then has gone into some kind of tight loop. No chunks
> >> have been found or balanced in the last 2 hours, and one kworker thread
> >> is pegged at 100%. There were no unusual or unexpected messages in the
> >> logs. Balance cancel has been waiting for the last 10 minutes without
> >> effect (as would be expected with the other symptoms).
> > 
> >    OK, an hour after I executed the cancel, and three hours after it
> > apparently jammed up, the cancel completed. I have restarted the balance
> > from the point it left off, and I'll see if it does something similar
> > again.

   I think it's having problems with metadata chunks -- either
directly with the fact that it's metadata, or with the fact that
there's tens of thousands of extents in the block group. The last
thing in my dmesg at the moment is:

[161779.883848] BTRFS info (device sdg2): relocating block group 12285846224896 flags 20

after 29 other successful block group relocations of type 17
(data/RAID-1). It's been on this one (type 20; metadata/RAID-1) for a
couple of hours at least. I'm going to let it run and see when (if) it
gets off this block group and moves on to a new one.

> I've looked through the patches I use since 3.18.0 and nothing too 
> obvious stood out (to me), except for several patches from Filipe for 
> fixes that revolve around the chunk reaper, which sound like they might 
> result in confused threads. Also moving to 3.18.2 probably won't hurt, 
> it's not like there are no bugs in the kernel itself -  I recently 
> managed to find a really wrong corner case in NFS that has gone unnoticed 
> since 3.16.

   I'll have a go with that when I next get the chance to reboot that
machine.

> Finally: instead of balancing everything at once maybe try a piecemeal 
> approach with the limit filter? Wrap it in a script, spread out the work 
> of balancing 3-5 chunks a time..

   Not sure that'll help. It's just hitting these occasional block
groups and coming to a grinding halt for hours before continuing.

   Hugo.

-- 
Hugo Mills             | You are not stuck in traffic: you are traffic
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: 65E74AC0          |                                    German ad campaign

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]