* Lost about 3TB [not found] <134025801.432834337.1507024250294.JavaMail.root@zimbra65-e11.priv.proxad.net> @ 2017-10-03 10:44 ` btrfs.fredo 2017-10-03 10:54 ` Hugo Mills 0 siblings, 1 reply; 7+ messages in thread From: btrfs.fredo @ 2017-10-03 10:44 UTC (permalink / raw) To: linux-btrfs Hi, I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). My BTRFS volume is mounted on /RAID01/. There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. ######> du -hs /RAID01/ 28T /RAID01/ If I sum up the result of : ######> find . -printf '%s\n' I also find 28TB. I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du on each file and the result is 28TB. OS : CentOS Linux release 7.3.1611 (Core) btrfs-progs v4.4.1 ######> ssm list ------------------------------------------------------------------------- Device Free Used Total Pool Mount point ------------------------------------------------------------------------- /dev/sda 36.39 TB PARTITIONED /dev/sda1 200.00 MB /boot/efi /dev/sda2 1.00 GB /boot /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 ------------------------------------------------------------------------- ------------------------------------------------------------------- Pool Type Devices Free Used Total ------------------------------------------------------------------- cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB ------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------- Volume Pool Volume size FS FS size Free Type Mount point --------------------------------------------------------------------------------------------------------------------- /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 /dev/sda1 200.00 MB vfat part /boot/efi /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot --------------------------------------------------------------------------------------------------------------------- ######> btrfs fi sh Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 Total devices 1 FS bytes used 31.48TiB devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 ######> btrfs fi df /RAID01/ Data, single: total=31.58TiB, used=31.44TiB System, DUP: total=8.00MiB, used=3.67MiB Metadata, DUP: total=38.00GiB, used=35.37GiB GlobalReserve, single: total=512.00MiB, used=0.00B I tried to repair it : ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 enabling repair mode Checking filesystem on /dev/mapper/lvm_pool-lvol001 UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 checking extents Fixed 0 roots. cache and super generation don't match, space cache will be invalidated checking fs roots checking csums checking root refs found 34600611349019 bytes used err is 0 total csum bytes: 33752513152 total tree bytes: 38037848064 total fs tree bytes: 583942144 total extent tree bytes: 653754368 btree space waste bytes: 2197658704 file data blocks allocated: 183716661284864 ?? what's this ?? referenced 30095956975616 = 27.3 TB !! Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB Overall: Device size: 36.32TiB Device allocated: 31.65TiB Device unallocated: 4.67TiB Device missing: 0.00B Used: 31.52TiB Free (estimated): 4.80TiB (min: 2.46TiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,single: Size:31.58TiB, Used:31.45TiB /dev/mapper/lvm_pool-lvol001 31.58TiB Metadata,DUP: Size:38.00GiB, Used:35.37GiB /dev/mapper/lvm_pool-lvol001 76.00GiB System,DUP: Size:8.00MiB, Used:3.69MiB /dev/mapper/lvm_pool-lvol001 16.00MiB Unallocated: /dev/mapper/lvm_pool-lvol001 4.67TiB The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") Code: file data blocks allocated: 183716661284864 ?? what's this ?? referenced 30095956975616 = 27.3 TB !! I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume What did I do wrong or am I missing ? Thanks in advance. Frederic Larive. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 10:44 ` Lost about 3TB btrfs.fredo @ 2017-10-03 10:54 ` Hugo Mills 2017-10-03 11:08 ` Timofey Titovets ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Hugo Mills @ 2017-10-03 10:54 UTC (permalink / raw) To: btrfs.fredo; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 6736 bytes --] On Tue, Oct 03, 2017 at 12:44:29PM +0200, btrfs.fredo@xoxy.net wrote: > Hi, > > I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! > > I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). > > My BTRFS volume is mounted on /RAID01/. > > There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. > > It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. > > ######> du -hs /RAID01/ > 28T /RAID01/ > > If I sum up the result of : ######> find . -printf '%s\n' > I also find 28TB. > > I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du > on each file and the result is 28TB. The conclusion here is that there are things that aren't being found by these processes. This is usually in the form of dot-files (but I think you've covered that case in what you did above) or snapshots/subvolumes outside the subvol you've mounted. What does "btrfs sub list -a /RAID01/" say? Also "grep /RAID01/ /proc/self/mountinfo"? There are other possibilities for missing space, but let's cover the obvious ones first. Hugo. > OS : CentOS Linux release 7.3.1611 (Core) > btrfs-progs v4.4.1 > > > ######> ssm list > > ------------------------------------------------------------------------- > Device Free Used Total Pool Mount point > ------------------------------------------------------------------------- > /dev/sda 36.39 TB PARTITIONED > /dev/sda1 200.00 MB /boot/efi > /dev/sda2 1.00 GB /boot > /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool > /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 > ------------------------------------------------------------------------- > ------------------------------------------------------------------- > Pool Type Devices Free Used Total > ------------------------------------------------------------------- > cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB > lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB > btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB > ------------------------------------------------------------------- > --------------------------------------------------------------------------------------------------------------------- > Volume Pool Volume size FS FS size Free Type Mount point > --------------------------------------------------------------------------------------------------------------------- > /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / > /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear > /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 > btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 > /dev/sda1 200.00 MB vfat part /boot/efi > /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot > --------------------------------------------------------------------------------------------------------------------- > > > ######> btrfs fi sh > > Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 > Total devices 1 FS bytes used 31.48TiB > devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 > > > > ######> btrfs fi df /RAID01/ > > Data, single: total=31.58TiB, used=31.44TiB > System, DUP: total=8.00MiB, used=3.67MiB > Metadata, DUP: total=38.00GiB, used=35.37GiB > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > I tried to repair it : > > > ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 > > enabling repair mode > Checking filesystem on /dev/mapper/lvm_pool-lvol001 > UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 > checking extents > Fixed 0 roots. > cache and super generation don't match, space cache will be invalidated > checking fs roots > checking csums > checking root refs > found 34600611349019 bytes used err is 0 > total csum bytes: 33752513152 > total tree bytes: 38037848064 > total fs tree bytes: 583942144 > total extent tree bytes: 653754368 > btree space waste bytes: 2197658704 > file data blocks allocated: 183716661284864 ?? what's this ?? > referenced 30095956975616 = 27.3 TB !! > > > > Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB > > Overall: > Device size: 36.32TiB > Device allocated: 31.65TiB > Device unallocated: 4.67TiB > Device missing: 0.00B > Used: 31.52TiB > Free (estimated): 4.80TiB (min: 2.46TiB) > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data,single: Size:31.58TiB, Used:31.45TiB > /dev/mapper/lvm_pool-lvol001 31.58TiB > > Metadata,DUP: Size:38.00GiB, Used:35.37GiB > /dev/mapper/lvm_pool-lvol001 76.00GiB > > System,DUP: Size:8.00MiB, Used:3.69MiB > /dev/mapper/lvm_pool-lvol001 16.00MiB > > Unallocated: > /dev/mapper/lvm_pool-lvol001 4.67TiB > The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") > Code: > file data blocks allocated: 183716661284864 ?? what's this ?? > referenced 30095956975616 = 27.3 TB !! > > > > I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. > > I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. > No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume > > > What did I do wrong or am I missing ? > > Thanks in advance. > Frederic Larive. > -- Hugo Mills | Beware geeks bearing GIFs hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 10:54 ` Hugo Mills @ 2017-10-03 11:08 ` Timofey Titovets 2017-10-03 12:44 ` Roman Mamedov 2017-10-03 15:45 ` fred.larive 2 siblings, 0 replies; 7+ messages in thread From: Timofey Titovets @ 2017-10-03 11:08 UTC (permalink / raw) To: Hugo Mills, btrfs.fredo, linux-btrfs 2017-10-03 13:54 GMT+03:00 Hugo Mills <hugo@carfax.org.uk>: > On Tue, Oct 03, 2017 at 12:44:29PM +0200, btrfs.fredo@xoxy.net wrote: >> Hi, >> >> I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! >> >> I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). >> >> My BTRFS volume is mounted on /RAID01/. >> >> There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. >> >> It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. >> >> ######> du -hs /RAID01/ >> 28T /RAID01/ >> >> If I sum up the result of : ######> find . -printf '%s\n' >> I also find 28TB. >> >> I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du >> on each file and the result is 28TB. > > The conclusion here is that there are things that aren't being > found by these processes. This is usually in the form of dot-files > (but I think you've covered that case in what you did above) or > snapshots/subvolumes outside the subvol you've mounted. > > What does "btrfs sub list -a /RAID01/" say? > Also "grep /RAID01/ /proc/self/mountinfo"? > > There are other possibilities for missing space, but let's cover > the obvious ones first. > > Hugo. > >> OS : CentOS Linux release 7.3.1611 (Core) >> btrfs-progs v4.4.1 >> >> >> ######> ssm list >> >> ------------------------------------------------------------------------- >> Device Free Used Total Pool Mount point >> ------------------------------------------------------------------------- >> /dev/sda 36.39 TB PARTITIONED >> /dev/sda1 200.00 MB /boot/efi >> /dev/sda2 1.00 GB /boot >> /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool >> /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 >> ------------------------------------------------------------------------- >> ------------------------------------------------------------------- >> Pool Type Devices Free Used Total >> ------------------------------------------------------------------- >> cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB >> lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB >> btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB >> ------------------------------------------------------------------- >> --------------------------------------------------------------------------------------------------------------------- >> Volume Pool Volume size FS FS size Free Type Mount point >> --------------------------------------------------------------------------------------------------------------------- >> /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / >> /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear >> /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 >> btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 >> /dev/sda1 200.00 MB vfat part /boot/efi >> /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot >> --------------------------------------------------------------------------------------------------------------------- >> >> >> ######> btrfs fi sh >> >> Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 >> Total devices 1 FS bytes used 31.48TiB >> devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 >> >> >> >> ######> btrfs fi df /RAID01/ >> >> Data, single: total=31.58TiB, used=31.44TiB >> System, DUP: total=8.00MiB, used=3.67MiB >> Metadata, DUP: total=38.00GiB, used=35.37GiB >> GlobalReserve, single: total=512.00MiB, used=0.00B >> >> >> >> I tried to repair it : >> >> >> ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 >> >> enabling repair mode >> Checking filesystem on /dev/mapper/lvm_pool-lvol001 >> UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 >> checking extents >> Fixed 0 roots. >> cache and super generation don't match, space cache will be invalidated >> checking fs roots >> checking csums >> checking root refs >> found 34600611349019 bytes used err is 0 >> total csum bytes: 33752513152 >> total tree bytes: 38037848064 >> total fs tree bytes: 583942144 >> total extent tree bytes: 653754368 >> btree space waste bytes: 2197658704 >> file data blocks allocated: 183716661284864 ?? what's this ?? >> referenced 30095956975616 = 27.3 TB !! >> >> >> >> Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB >> >> Overall: >> Device size: 36.32TiB >> Device allocated: 31.65TiB >> Device unallocated: 4.67TiB >> Device missing: 0.00B >> Used: 31.52TiB >> Free (estimated): 4.80TiB (min: 2.46TiB) >> Data ratio: 1.00 >> Metadata ratio: 2.00 >> Global reserve: 512.00MiB (used: 0.00B) >> >> Data,single: Size:31.58TiB, Used:31.45TiB >> /dev/mapper/lvm_pool-lvol001 31.58TiB >> >> Metadata,DUP: Size:38.00GiB, Used:35.37GiB >> /dev/mapper/lvm_pool-lvol001 76.00GiB >> >> System,DUP: Size:8.00MiB, Used:3.69MiB >> /dev/mapper/lvm_pool-lvol001 16.00MiB >> >> Unallocated: >> /dev/mapper/lvm_pool-lvol001 4.67TiB >> The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") >> Code: >> file data blocks allocated: 183716661284864 ?? what's this ?? >> referenced 30095956975616 = 27.3 TB !! >> >> >> >> I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. >> >> I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. >> No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume >> >> >> What did I do wrong or am I missing ? >> >> Thanks in advance. >> Frederic Larive. >> > > -- > Hugo Mills | Beware geeks bearing GIFs > hugo@... carfax.org.uk | > http://carfax.org.uk/ | > PGP: E2AB1DE4 | If your storage not write-once, then that can be simple extent book keeping. i.e. as proof, i create empty fs, create 128KiB file Then write 2*4KiB random sectors, and get that: ~ filefrag -v /mnt/128KiB Filesystem type is: 9123683e File size of /mnt/128KiB is 131072 (32 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2: 3104.. 3106: 3: 1: 3.. 3: 3088.. 3088: 1: 3107: 2: 4.. 18: 3108.. 3122: 15: 3089: 3: 19.. 19: 3072.. 3072: 1: 3123: 4: 20.. 31: 3124.. 3135: 12: 3073: last,eof /mnt/128KiB: 5 extents found ~ btrfs fi df /mnt Data, single: total=8.00MiB, used=200.00KiB System, single: total=4.00MiB, used=16.00KiB Metadata, single: total=120.00MiB, used=112.00KiB GlobalReserve, single: total=16.00MiB, used=0.00B On empty fs 64KiB used by something (i'm not sure that use it) 200 - 64 = 136 KiB 136 - 128 = 8 KiB 8/4 = 2 writes So on your FS that just can be fragmented space. -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 10:54 ` Hugo Mills 2017-10-03 11:08 ` Timofey Titovets @ 2017-10-03 12:44 ` Roman Mamedov 2017-10-03 15:45 ` fred.larive 2 siblings, 0 replies; 7+ messages in thread From: Roman Mamedov @ 2017-10-03 12:44 UTC (permalink / raw) To: Hugo Mills; +Cc: btrfs.fredo, linux-btrfs On Tue, 3 Oct 2017 10:54:05 +0000 Hugo Mills <hugo@carfax.org.uk> wrote: > There are other possibilities for missing space, but let's cover > the obvious ones first. One more obvious thing would be files that are deleted, but still kept open by some app (possibly even from network, via NFS or SMB!). @Frederic, did you try rebooting the system? -- With respect, Roman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 10:54 ` Hugo Mills 2017-10-03 11:08 ` Timofey Titovets 2017-10-03 12:44 ` Roman Mamedov @ 2017-10-03 15:45 ` fred.larive 2017-10-03 16:00 ` Hugo Mills 2 siblings, 1 reply; 7+ messages in thread From: fred.larive @ 2017-10-03 15:45 UTC (permalink / raw) To: Hugo Mills - hugo@carfax.org.uk; +Cc: linux-btrfs, btrfs fredo Hi, > What does "btrfs sub list -a /RAID01/" say? Nothing (no lines displayed) > Also "grep /RAID01/ /proc/self/mountinfo"? Nothing (no lines displayed) Also server has been rebooted many times and no process has left "deleted open files" on the volume (lsof...). Fred. ----- Mail original ----- De: "Hugo Mills - hugo@carfax.org.uk" <btrfs.fredo.d1c3ddb588.hugo#carfax.org.uk@ob.0sg.net> À: "btrfs fredo" <btrfs.fredo@xoxy.net> Cc: linux-btrfs@vger.kernel.org Envoyé: Mardi 3 Octobre 2017 12:54:05 Objet: Re: Lost about 3TB On Tue, Oct 03, 2017 at 12:44:29PM +0200, btrfs.fredo@xoxy.net wrote: > Hi, > > I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! > > I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). > > My BTRFS volume is mounted on /RAID01/. > > There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. > > It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. > > ######> du -hs /RAID01/ > 28T /RAID01/ > > If I sum up the result of : ######> find . -printf '%s\n' > I also find 28TB. > > I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du > on each file and the result is 28TB. The conclusion here is that there are things that aren't being found by these processes. This is usually in the form of dot-files (but I think you've covered that case in what you did above) or snapshots/subvolumes outside the subvol you've mounted. What does "btrfs sub list -a /RAID01/" say? Also "grep /RAID01/ /proc/self/mountinfo"? There are other possibilities for missing space, but let's cover the obvious ones first. Hugo. > OS : CentOS Linux release 7.3.1611 (Core) > btrfs-progs v4.4.1 > > > ######> ssm list > > ------------------------------------------------------------------------- > Device Free Used Total Pool Mount point > ------------------------------------------------------------------------- > /dev/sda 36.39 TB PARTITIONED > /dev/sda1 200.00 MB /boot/efi > /dev/sda2 1.00 GB /boot > /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool > /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 > ------------------------------------------------------------------------- > ------------------------------------------------------------------- > Pool Type Devices Free Used Total > ------------------------------------------------------------------- > cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB > lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB > btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB > ------------------------------------------------------------------- > --------------------------------------------------------------------------------------------------------------------- > Volume Pool Volume size FS FS size Free Type Mount point > --------------------------------------------------------------------------------------------------------------------- > /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / > /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear > /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 > btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 > /dev/sda1 200.00 MB vfat part /boot/efi > /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot > --------------------------------------------------------------------------------------------------------------------- > > > ######> btrfs fi sh > > Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 > Total devices 1 FS bytes used 31.48TiB > devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 > > > > ######> btrfs fi df /RAID01/ > > Data, single: total=31.58TiB, used=31.44TiB > System, DUP: total=8.00MiB, used=3.67MiB > Metadata, DUP: total=38.00GiB, used=35.37GiB > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > I tried to repair it : > > > ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 > > enabling repair mode > Checking filesystem on /dev/mapper/lvm_pool-lvol001 > UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 > checking extents > Fixed 0 roots. > cache and super generation don't match, space cache will be invalidated > checking fs roots > checking csums > checking root refs > found 34600611349019 bytes used err is 0 > total csum bytes: 33752513152 > total tree bytes: 38037848064 > total fs tree bytes: 583942144 > total extent tree bytes: 653754368 > btree space waste bytes: 2197658704 > file data blocks allocated: 183716661284864 ?? what's this ?? > referenced 30095956975616 = 27.3 TB !! > > > > Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB > > Overall: > Device size: 36.32TiB > Device allocated: 31.65TiB > Device unallocated: 4.67TiB > Device missing: 0.00B > Used: 31.52TiB > Free (estimated): 4.80TiB (min: 2.46TiB) > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data,single: Size:31.58TiB, Used:31.45TiB > /dev/mapper/lvm_pool-lvol001 31.58TiB > > Metadata,DUP: Size:38.00GiB, Used:35.37GiB > /dev/mapper/lvm_pool-lvol001 76.00GiB > > System,DUP: Size:8.00MiB, Used:3.69MiB > /dev/mapper/lvm_pool-lvol001 16.00MiB > > Unallocated: > /dev/mapper/lvm_pool-lvol001 4.67TiB > The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") > Code: > file data blocks allocated: 183716661284864 ?? what's this ?? > referenced 30095956975616 = 27.3 TB !! > > > > I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. > > I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. > No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume > > > What did I do wrong or am I missing ? > > Thanks in advance. > Frederic Larive. > -- Hugo Mills | Beware geeks bearing GIFs hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 15:45 ` fred.larive @ 2017-10-03 16:00 ` Hugo Mills 2017-10-04 12:43 ` fred.larive 0 siblings, 1 reply; 7+ messages in thread From: Hugo Mills @ 2017-10-03 16:00 UTC (permalink / raw) To: fred.larive; +Cc: Hugo Mills - hugo@carfax.org.uk, linux-btrfs, btrfs fredo [-- Attachment #1: Type: text/plain, Size: 8951 bytes --] On Tue, Oct 03, 2017 at 05:45:54PM +0200, fred.larive@free.fr wrote: > Hi, > > > > What does "btrfs sub list -a /RAID01/" say? > Nothing (no lines displayed) > > > Also "grep /RAID01/ /proc/self/mountinfo"? > Nothing (no lines displayed) > > > Also server has been rebooted many times and no process has left "deleted open files" on the volume (lsof...). OK. The second command (the grep) was incorrect -- I should have omitted the slashes. However, it doesn't matter too much, because the first command indicates that you don't have any subvolumes or snapshots anyway. This means that you're probably looking at the kind of issue Timofey mentioned in his mail, where writes into the middle of an existing extent don't free up the overwritten data. This is most likely to happen on database or VM files, but could happen on others, depending on the application and how it uses files. Since you don't seem to have any snapshots, I _think_ you can deal with the issue most easily by defragmenting the affected files. It's worth just getting a second opinion on this one before you try it for the whole FS. I'm not 100% sure about what defrag will do in this case, and there are some people round here who have investigated the behaviour of partially-overwritten extents in more detail than I have. Hugo. > Fred. > > > ----- Mail original ----- > De: "Hugo Mills - hugo@carfax.org.uk" <btrfs.fredo.d1c3ddb588.hugo#carfax.org.uk@ob.0sg.net> > À: "btrfs fredo" <btrfs.fredo@xoxy.net> > Cc: linux-btrfs@vger.kernel.org > Envoyé: Mardi 3 Octobre 2017 12:54:05 > Objet: Re: Lost about 3TB > > On Tue, Oct 03, 2017 at 12:44:29PM +0200, btrfs.fredo@xoxy.net wrote: > > Hi, > > > > I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! > > > > I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). > > > > My BTRFS volume is mounted on /RAID01/. > > > > There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. > > > > It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. > > > > ######> du -hs /RAID01/ > > 28T /RAID01/ > > > > If I sum up the result of : ######> find . -printf '%s\n' > > I also find 28TB. > > > > I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du > > on each file and the result is 28TB. > > The conclusion here is that there are things that aren't being > found by these processes. This is usually in the form of dot-files > (but I think you've covered that case in what you did above) or > snapshots/subvolumes outside the subvol you've mounted. > > What does "btrfs sub list -a /RAID01/" say? > Also "grep /RAID01/ /proc/self/mountinfo"? > > There are other possibilities for missing space, but let's cover > the obvious ones first. > > Hugo. > > > OS : CentOS Linux release 7.3.1611 (Core) > > btrfs-progs v4.4.1 > > > > > > ######> ssm list > > > > ------------------------------------------------------------------------- > > Device Free Used Total Pool Mount point > > ------------------------------------------------------------------------- > > /dev/sda 36.39 TB PARTITIONED > > /dev/sda1 200.00 MB /boot/efi > > /dev/sda2 1.00 GB /boot > > /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool > > /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 > > ------------------------------------------------------------------------- > > ------------------------------------------------------------------- > > Pool Type Devices Free Used Total > > ------------------------------------------------------------------- > > cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB > > lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB > > btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB > > ------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------------------------------- > > Volume Pool Volume size FS FS size Free Type Mount point > > --------------------------------------------------------------------------------------------------------------------- > > /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / > > /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear > > /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 > > btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 > > /dev/sda1 200.00 MB vfat part /boot/efi > > /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot > > --------------------------------------------------------------------------------------------------------------------- > > > > > > ######> btrfs fi sh > > > > Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 > > Total devices 1 FS bytes used 31.48TiB > > devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 > > > > > > > > ######> btrfs fi df /RAID01/ > > > > Data, single: total=31.58TiB, used=31.44TiB > > System, DUP: total=8.00MiB, used=3.67MiB > > Metadata, DUP: total=38.00GiB, used=35.37GiB > > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > > > > > I tried to repair it : > > > > > > ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 > > > > enabling repair mode > > Checking filesystem on /dev/mapper/lvm_pool-lvol001 > > UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 > > checking extents > > Fixed 0 roots. > > cache and super generation don't match, space cache will be invalidated > > checking fs roots > > checking csums > > checking root refs > > found 34600611349019 bytes used err is 0 > > total csum bytes: 33752513152 > > total tree bytes: 38037848064 > > total fs tree bytes: 583942144 > > total extent tree bytes: 653754368 > > btree space waste bytes: 2197658704 > > file data blocks allocated: 183716661284864 ?? what's this ?? > > referenced 30095956975616 = 27.3 TB !! > > > > > > > > Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB > > > > Overall: > > Device size: 36.32TiB > > Device allocated: 31.65TiB > > Device unallocated: 4.67TiB > > Device missing: 0.00B > > Used: 31.52TiB > > Free (estimated): 4.80TiB (min: 2.46TiB) > > Data ratio: 1.00 > > Metadata ratio: 2.00 > > Global reserve: 512.00MiB (used: 0.00B) > > > > Data,single: Size:31.58TiB, Used:31.45TiB > > /dev/mapper/lvm_pool-lvol001 31.58TiB > > > > Metadata,DUP: Size:38.00GiB, Used:35.37GiB > > /dev/mapper/lvm_pool-lvol001 76.00GiB > > > > System,DUP: Size:8.00MiB, Used:3.69MiB > > /dev/mapper/lvm_pool-lvol001 16.00MiB > > > > Unallocated: > > /dev/mapper/lvm_pool-lvol001 4.67TiB > > The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") > > Code: > > file data blocks allocated: 183716661284864 ?? what's this ?? > > referenced 30095956975616 = 27.3 TB !! > > > > > > > > I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. > > > > I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. > > No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume > > > > > > What did I do wrong or am I missing ? > > > > Thanks in advance. > > Frederic Larive. > > > -- Hugo Mills | Well, sir, the floor is yours. But remember, the hugo@... carfax.org.uk | roof is ours! http://carfax.org.uk/ | PGP: E2AB1DE4 | The Goons [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Lost about 3TB 2017-10-03 16:00 ` Hugo Mills @ 2017-10-04 12:43 ` fred.larive 0 siblings, 0 replies; 7+ messages in thread From: fred.larive @ 2017-10-04 12:43 UTC (permalink / raw) To: Hugo Mills; +Cc: Hugo Mills - hugo@carfax.org.uk, linux-btrfs, btrfs fredo Hi, thanks all for your fast support. I did some stats about fragmentation (I didn't do it before because, I wrongly thought it was nearly the same as chunk balance). As Timofey suspected I have some big files (5TB) heavily fragmented. The software writting in these files is a backup software called Veeam, working at block level of virtual machines : it create an image of the virtual disk and the incremental backup only takes care about newly changed blocks. But we use it each month to "synthetically" create monthly backup restore point. So yes, most of write is sparse write of small blocks, this server is over a WAN link and Veeam is incredbly robust allowing us to break a backup job and continue it later (but taking in account new source blocks) without resending the whole file, the result is then fragmented (I traced it during restore job and can see that it is able to find needed blocks on every received disks images even in broken ones !). To sum up : Each files extent average size is 15MB, but at least 30MB (up to 60MB) since I balanced the volume. But I have four 5TB files with average extent size being between 3 to 7 MB split over about 1 to 2 million extents. I tried defrag on some 30000 extents files dividing the extent number by two and freeing some used space. I'm now defragging the whole volume with "-t 128M", permitting to free 200GB in one hour. I think you hit my problem. I'll keep you in touch, I think tomorrow. Thank's again. Fred. ----- Mail original ----- De: "Hugo Mills" <hugo@carfax.org.uk> À: "fred larive" <fred.larive@free.fr> Cc: "Hugo Mills - hugo@carfax.org.uk" <btrfs.fredo.d1c3ddb588.hugo#carfax.org.uk@ob.0sg.net>, linux-btrfs@vger.kernel.org, "btrfs fredo" <btrfs.fredo@xoxy.net> Envoyé: Mardi 3 Octobre 2017 18:00:04 Objet: Re: Lost about 3TB On Tue, Oct 03, 2017 at 05:45:54PM +0200, fred.larive@free.fr wrote: > Hi, > > > > What does "btrfs sub list -a /RAID01/" say? > Nothing (no lines displayed) > > > Also "grep /RAID01/ /proc/self/mountinfo"? > Nothing (no lines displayed) > > > Also server has been rebooted many times and no process has left "deleted open files" on the volume (lsof...). OK. The second command (the grep) was incorrect -- I should have omitted the slashes. However, it doesn't matter too much, because the first command indicates that you don't have any subvolumes or snapshots anyway. This means that you're probably looking at the kind of issue Timofey mentioned in his mail, where writes into the middle of an existing extent don't free up the overwritten data. This is most likely to happen on database or VM files, but could happen on others, depending on the application and how it uses files. Since you don't seem to have any snapshots, I _think_ you can deal with the issue most easily by defragmenting the affected files. It's worth just getting a second opinion on this one before you try it for the whole FS. I'm not 100% sure about what defrag will do in this case, and there are some people round here who have investigated the behaviour of partially-overwritten extents in more detail than I have. Hugo. > Fred. > > > ----- Mail original ----- > De: "Hugo Mills - hugo@carfax.org.uk" <btrfs.fredo.d1c3ddb588.hugo#carfax.org.uk@ob.0sg.net> > À: "btrfs fredo" <btrfs.fredo@xoxy.net> > Cc: linux-btrfs@vger.kernel.org > Envoyé: Mardi 3 Octobre 2017 12:54:05 > Objet: Re: Lost about 3TB > > On Tue, Oct 03, 2017 at 12:44:29PM +0200, btrfs.fredo@xoxy.net wrote: > > Hi, > > > > I can't figure out were 3TB on a 36 TB BTRFS volume (on LVM) are gone ! > > > > I know BTRFS can be tricky when speaking about space usage when using many physical drives in a RAID setup, but my conf is a very simple BTRFS volume without RAID(single Data type) using the whole disk (perhaps did I do something wrong with the LVM setup ?). > > > > My BTRFS volume is mounted on /RAID01/. > > > > There's only one folder in /RAID01/ shared with Samba, Windows also see a total of 28 TB used. > > > > It only contains 443 files (big backup files created by Veeam), most of the file size is greater than 1GB and be be up to 5TB. > > > > ######> du -hs /RAID01/ > > 28T /RAID01/ > > > > If I sum up the result of : ######> find . -printf '%s\n' > > I also find 28TB. > > > > I extracted btrfs binary from rpm version v4.9.1 and used ######> btrfs fi du > > on each file and the result is 28TB. > > The conclusion here is that there are things that aren't being > found by these processes. This is usually in the form of dot-files > (but I think you've covered that case in what you did above) or > snapshots/subvolumes outside the subvol you've mounted. > > What does "btrfs sub list -a /RAID01/" say? > Also "grep /RAID01/ /proc/self/mountinfo"? > > There are other possibilities for missing space, but let's cover > the obvious ones first. > > Hugo. > > > OS : CentOS Linux release 7.3.1611 (Core) > > btrfs-progs v4.4.1 > > > > > > ######> ssm list > > > > ------------------------------------------------------------------------- > > Device Free Used Total Pool Mount point > > ------------------------------------------------------------------------- > > /dev/sda 36.39 TB PARTITIONED > > /dev/sda1 200.00 MB /boot/efi > > /dev/sda2 1.00 GB /boot > > /dev/sda3 0.00 KB 36.32 TB 36.32 TB lvm_pool > > /dev/sda4 0.00 KB 54.00 GB 54.00 GB cl_xxx-xxxamrepo-01 > > ------------------------------------------------------------------------- > > ------------------------------------------------------------------- > > Pool Type Devices Free Used Total > > ------------------------------------------------------------------- > > cl_xxx-xxxamrepo-01 lvm 1 0.00 KB 54.00 GB 54.00 GB > > lvm_pool lvm 1 0.00 KB 36.32 TB 36.32 TB > > btrfs_lvm_pool-lvol001 btrfs 1 4.84 TB 36.32 TB 36.32 TB > > ------------------------------------------------------------------- > > --------------------------------------------------------------------------------------------------------------------- > > Volume Pool Volume size FS FS size Free Type Mount point > > --------------------------------------------------------------------------------------------------------------------- > > /dev/cl_xxx-xxxamrepo-01/root cl_xxx-xxxamrepo-01 50.00 GB xfs 49.97 GB 48.50 GB linear / > > /dev/cl_xxx-xxxamrepo-01/swap cl_xxx-xxxamrepo-01 4.00 GB linear > > /dev/lvm_pool/lvol001 lvm_pool 36.32 TB linear /RAID01 > > btrfs_lvm_pool-lvol001 btrfs_lvm_pool-lvol001 36.32 TB btrfs 36.32 TB 4.84 TB btrfs /RAID01 > > /dev/sda1 200.00 MB vfat part /boot/efi > > /dev/sda2 1.00 GB xfs 1015.00 MB 882.54 MB part /boot > > --------------------------------------------------------------------------------------------------------------------- > > > > > > ######> btrfs fi sh > > > > Label: none uuid: df7ce232-056a-4c27-bde4-6f785d5d9f68 > > Total devices 1 FS bytes used 31.48TiB > > devid 1 size 36.32TiB used 31.66TiB path /dev/mapper/lvm_pool-lvol001 > > > > > > > > ######> btrfs fi df /RAID01/ > > > > Data, single: total=31.58TiB, used=31.44TiB > > System, DUP: total=8.00MiB, used=3.67MiB > > Metadata, DUP: total=38.00GiB, used=35.37GiB > > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > > > > > I tried to repair it : > > > > > > ######> btrfs check --repair -p /dev/mapper/lvm_pool-lvol001 > > > > enabling repair mode > > Checking filesystem on /dev/mapper/lvm_pool-lvol001 > > UUID: df7ce232-056a-4c27-bde4-6f785d5d9f68 > > checking extents > > Fixed 0 roots. > > cache and super generation don't match, space cache will be invalidated > > checking fs roots > > checking csums > > checking root refs > > found 34600611349019 bytes used err is 0 > > total csum bytes: 33752513152 > > total tree bytes: 38037848064 > > total fs tree bytes: 583942144 > > total extent tree bytes: 653754368 > > btree space waste bytes: 2197658704 > > file data blocks allocated: 183716661284864 ?? what's this ?? > > referenced 30095956975616 = 27.3 TB !! > > > > > > > > Tried the "new usage" display but the problem is the same : 31 TB used but total file size is 28TB > > > > Overall: > > Device size: 36.32TiB > > Device allocated: 31.65TiB > > Device unallocated: 4.67TiB > > Device missing: 0.00B > > Used: 31.52TiB > > Free (estimated): 4.80TiB (min: 2.46TiB) > > Data ratio: 1.00 > > Metadata ratio: 2.00 > > Global reserve: 512.00MiB (used: 0.00B) > > > > Data,single: Size:31.58TiB, Used:31.45TiB > > /dev/mapper/lvm_pool-lvol001 31.58TiB > > > > Metadata,DUP: Size:38.00GiB, Used:35.37GiB > > /dev/mapper/lvm_pool-lvol001 76.00GiB > > > > System,DUP: Size:8.00MiB, Used:3.69MiB > > /dev/mapper/lvm_pool-lvol001 16.00MiB > > > > Unallocated: > > /dev/mapper/lvm_pool-lvol001 4.67TiB > > The only btrfs tool speaking about 28TB is btrfs check (but I'm not sure if it's bytes because it speaks about "referenced blocks" and I don't understand the meaning of "file data blocks allocated") > > Code: > > file data blocks allocated: 183716661284864 ?? what's this ?? > > referenced 30095956975616 = 27.3 TB !! > > > > > > > > I also used the verbose option of https://github.com/knorrie/btrfs-heatmap/ to sum up the total size of all DATA EXTENT and found 32TB. > > > > I did scrub, balance up to -dusage=90 (and also dusage=0) and ended up with 32TB used. > > No snasphots nor subvolumes nor TB hidden under the mount point after unmounting the BTRFS volume > > > > > > What did I do wrong or am I missing ? > > > > Thanks in advance. > > Frederic Larive. > > > -- Hugo Mills | Well, sir, the floor is yours. But remember, the hugo@... carfax.org.uk | roof is ours! http://carfax.org.uk/ | PGP: E2AB1DE4 | The Goons ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-10-04 12:43 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <134025801.432834337.1507024250294.JavaMail.root@zimbra65-e11.priv.proxad.net>
2017-10-03 10:44 ` Lost about 3TB btrfs.fredo
2017-10-03 10:54 ` Hugo Mills
2017-10-03 11:08 ` Timofey Titovets
2017-10-03 12:44 ` Roman Mamedov
2017-10-03 15:45 ` fred.larive
2017-10-03 16:00 ` Hugo Mills
2017-10-04 12:43 ` fred.larive
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.