Re: new database files not compressed

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

On Fri, Sep 04, 2020 at 06:07:32PM +1000, Hamish Moffatt wrote:
> On 4/9/20 5:44 am, Zygo Blaxell wrote:
> > On Thu, Sep 03, 2020 at 10:53:23PM +1000, Hamish Moffatt wrote:
> > 
> > > I recompiled Firebird with fallocate disabled (it has a fallback for
> > > non-linux OSs), and now I have compressed database files.
> > > 
> > > It may be that de-duplication suits my application better anyway. Will
> > > compsize tell me how much space is being saved by de-duplication, or is
> > > there another way to find out?
> > Compsize reports "Uncompressed" and "Referenced" columns.  "Uncompressed"
> > is the physical size of the uncompressed data (i.e. how many bytes
> > you would need to hold all of the extents on disk without compression
> > but with dedupe).  "Referenced" is the logical size of the data, after
> > counting each reference (i.e. how many bytes you would need to hold all
> > of the data without compression or dedupe).
> > 
> > The "none" and "zstd" rows will tell you how much dedupe you're getting
> > on uncompressed and compressed extents separately.
> 
> 
> Great, I have it bees running and I see the deduplication in compsize as you
> said.
> 
> What is the appropriate place to ask question about bees - here, github or
> elsewhere?

I'm in all three places, though I might miss it if it's only posted to
the linux-btrfs list.  If you need to send a log file and you don't want it
to be fully public, there's an email address in the bees README.

> I added some files, restarted bees and it ran a deduplication, but then I
> added some more files (8 hours ago) and there's been some regularly logging
> but the new files haven't been deduplicated.

Information we'd need for this:

	- kernel version you're running

	- bees log, preferably at a level high enough to see the individual
	dedupe ops

	- btrfs dump-tree of the subvol trees containing the files

or a small reproducer.

If there's something like this in the log:

	2020-09-05 04:03:13 4464.4468<5> crawl_5: WORKAROUND: toxic address: addr = 0x544eb000, sys_usage_delta = 0.136, user_usage_delta = 0.02, rt_age = 0.251574, refs 1
	2020-09-05 04:03:13 4464.4468<4> crawl_5: WORKAROUND: discovered toxic match at found_addr 0x544eb000 matching bbd BeesBlockData { 4K 0x1db000 fd = 9 '/try/31264-2', address = 0x54823000, hash = 0x9a2052ab5fe19ae, data[4096] }

then it's bees detecting btrfs taking too long to do an extent lookup, and
blacklisting the extent to avoid crippling performance problems.  On 5.7
it seems to be happening a lot...ironic, given that 5.7 has much faster
backref code.


> Hamish
> 
>