From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A9A7C43381 for ; Fri, 15 Mar 2019 18:01:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D2D092186A for ; Fri, 15 Mar 2019 18:01:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726618AbfCOSBe (ORCPT ); Fri, 15 Mar 2019 14:01:34 -0400 Received: from james.kirk.hungrycats.org ([174.142.39.145]:39170 "EHLO james.kirk.hungrycats.org" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725956AbfCOSBe (ORCPT ); Fri, 15 Mar 2019 14:01:34 -0400 Received: by james.kirk.hungrycats.org (Postfix, from userid 1002) id CC1A4263A50; Fri, 15 Mar 2019 14:01:26 -0400 (EDT) Date: Fri, 15 Mar 2019 14:01:23 -0400 From: Zygo Blaxell To: Jakub =?iso-8859-1?Q?Hus=E1k?= Cc: linux-btrfs@vger.kernel.org Subject: Re: Balancing raid5 after adding another disk does not move/use any data on it Message-ID: <20190315180123.GJ9995@hungrycats.org> References: <7a713010-5db6-2627-2593-8e13092868b1@husak.pro> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MrRUTeZlqqNo1jQ9" Content-Disposition: inline In-Reply-To: <7a713010-5db6-2627-2593-8e13092868b1@husak.pro> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org --MrRUTeZlqqNo1jQ9 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 13, 2019 at 11:11:02PM +0100, Jakub Hus=E1k wrote: > Sorry, fighting with this technology called "email" :) >=20 >=20 > Hopefully better wrapped outputs: >=20 > On 13. 03. 19 22:58, Jakub Hus=E1k wrote: >=20 >=20 > > Hi, > >=20 > > I added another disk to my 3-disk raid5 and ran a balance command. After > > few hours I looked to output of `fi usage` to see that no data are being > > used on the new disk. I got the same result even when balancing my raid5 > > data or metadata. > >=20 > > Next I tried to convert my raid5 metadata to raid1 (a good idea anyway) > > and the new disk started to fill immediately (even though it received > > the whole amount of metadata with replicas being spread among the other > > drives, instead of being really "balanced". I know why this happened, I > > don't like it but I can live with it, let's not go off topic here :)). > >=20 > > Now my usage output looks like this: > >=20 > # btrfs filesystem usage=A0=A0 /mnt/data1 > WARNING: RAID56 detected, not implemented > Overall: > =A0=A0=A0 Device size:=A0=A0=A0 =A0=A0=A0 =A0 10.91TiB > =A0=A0=A0 Device allocated:=A0=A0=A0 =A0=A0=A0 =A0316.12GiB > =A0=A0=A0 Device unallocated:=A0=A0=A0 =A0=A0=A0 =A0 10.61TiB > =A0=A0=A0 Device missing:=A0=A0=A0 =A0=A0=A0 =A0=A0=A0=A0 0.00B > =A0=A0=A0 Used:=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0 58.86GiB > =A0=A0=A0 Free (estimated):=A0=A0=A0 =A0=A0=A0 =A0=A0=A0=A0 0.00B=A0=A0= =A0 (min: 8.00EiB) > =A0=A0=A0 Data ratio:=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0=A0=A0 0.00 > =A0=A0=A0 Metadata ratio:=A0=A0=A0 =A0=A0=A0 =A0=A0=A0=A0=A0 2.00 > =A0=A0=A0 Global reserve:=A0=A0=A0 =A0=A0=A0 =A0512.00MiB=A0=A0=A0 (used:= 0.00B) >=20 > Data,RAID5: Size:4.59TiB, Used:4.06TiB > =A0=A0 /dev/mapper/crypt-sdb=A0=A0=A0 =A0=A0 2.29TiB > =A0=A0 /dev/mapper/crypt-sdc=A0=A0=A0 =A0=A0 2.29TiB > =A0=A0 /dev/mapper/crypt-sde=A0=A0=A0 =A0=A0 2.29TiB >=20 > Metadata,RAID1: Size:158.00GiB, Used:29.43GiB > =A0=A0 /dev/mapper/crypt-sdb=A0=A0=A0 =A0 53.00GiB > =A0=A0 /dev/mapper/crypt-sdc=A0=A0=A0 =A0 53.00GiB > =A0=A0 /dev/mapper/crypt-sdd=A0=A0=A0 =A0158.00GiB > =A0=A0 /dev/mapper/crypt-sde=A0=A0=A0 =A0 52.00GiB >=20 > System,RAID1: Size:64.00MiB, Used:528.00KiB > =A0=A0 /dev/mapper/crypt-sdc=A0=A0=A0 =A0 32.00MiB > =A0=A0 /dev/mapper/crypt-sdd=A0=A0=A0 =A0 64.00MiB > =A0=A0 /dev/mapper/crypt-sde=A0=A0=A0 =A0 32.00MiB >=20 > Unallocated: > =A0=A0 /dev/mapper/crypt-sdb=A0=A0=A0 =A0393.04GiB > =A0=A0 /dev/mapper/crypt-sdc=A0=A0=A0 =A0393.01GiB > =A0=A0 /dev/mapper/crypt-sdd=A0=A0=A0 =A0=A0 2.57TiB > =A0=A0 /dev/mapper/crypt-sde=A0=A0=A0 =A0394.01GiB >=20 > >=20 > > I'm now running `fi balance -dusage=3D10` (and rising the usage limit).= I > > can see that the unallocated space is rising as it's freeing the little > > used chunks but still no data are being stored on the new disk. That is exactly what is happening: you are moving tiny amounts of data into existing big empty spaces, so no new chunk allocations (which should use the new drive) are happening. You have 470GB of data allocated but not used, so you have up to 235 block groups to fill before the new drive gets any data. Also note that you always have to do a full data balance when adding devices to raid5 in order to make use of all the space, so you might as well get started on that now. It'll take a while. 'btrfs balance start -dstripes=3D1..3 /mnt/data1' will work for this case. > > I it some bug? Is `fi usage` not showing me something (as it states > > "WARNING: RAID56 detected, not implemented")?=20 The warning just means the fields in the 'fi usage' output header, like "Free (estimate)", have bogus values because they're not computed correctly. > > Or is there just too much > > free space on the first set of disks that the balancing is not bothering > > moving any data? Yes. ;) > > If so, shouldn't it be really balancing (spreading) the data among all > > the drives to use all the IOPS capacity, even when the raid5 redundancy > > constraint is currently satisfied? btrfs divides the disks into chunks first, then spreads the data across the chunks. The chunk allocation behavior spreads chunks across all the disks. When you are adding a disk to raid5, you have to redistribute all the old data across all the disks to get balanced IOPS and space usage, hence the full balance requirement. If you don't do a full balance, it will eventually allocate data on all disks, but it will run out of space on sdb, sdc, and sde first, and then be unable to use the remaining 2TB+ on sdd. > >=20 > #=A0 uname -a > Linux storage 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1 > (2019-02-07) x86_64 GNU/Linux > #=A0=A0 btrfs --version > btrfs-progs v4.17 > #=A0 btrfs fi show > Label: none=A0 uuid: xxxxxxxxxxxxxxxxx > =A0=A0=A0 Total devices 4 FS bytes used 4.09TiB > =A0=A0=A0 devid=A0=A0=A0 2 size 2.73TiB used 2.34TiB path /dev/mapper/cry= pt-sdc > =A0=A0=A0 devid=A0=A0=A0 3 size 2.73TiB used 2.34TiB path /dev/mapper/cry= pt-sdb > =A0=A0=A0 devid=A0=A0=A0 4 size 2.73TiB used 2.34TiB path /dev/mapper/cry= pt-sde > =A0=A0=A0 devid=A0=A0=A0 5 size 2.73TiB used 158.06GiB path /dev/mapper/c= rypt-sdd >=20 > #=A0=A0 btrfs fi df . > Data, RAID5: total=3D4.59TiB, used=3D4.06TiB > System, RAID1: total=3D64.00MiB, used=3D528.00KiB > Metadata, RAID1: total=3D158.00GiB, used=3D29.43GiB > GlobalReserve, single: total=3D512.00MiB, used=3D0.00B >=20 > > Thanks > >=20 > > Jakub > >=20 --MrRUTeZlqqNo1jQ9 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EABECAB0WIQSnOVjcfGcC/+em7H2B+YsaVrMbnAUCXIvobwAKCRCB+YsaVrMb nMGdAJ9V0F8DICkkLhPyp3Ce67PkwJYVEQCgiIPnQMmmJYSzInjUjDVLRecrc0U= =aE0s -----END PGP SIGNATURE----- --MrRUTeZlqqNo1jQ9--