Bug with --buffer_compress_percentage

Bug with --buffer_compress_percentage

[Matthew Eaton]

I noticed that sometimes using --buffer_compress_percentage would not
result in the desired compression.  After running some tests I found
that only some block sizes are affected.

512b, 4K, 8K, 16K, and 32K seem to work as expected.

64K, 128K, 256K, 512K, and 1024K exhibited the bug.  I have not tested
beyond 1024K.

Here is the test script I used only changing block size for each test.

for i in {0..100}; do
  fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
--iodepth=32 --size=512m --refill_buffers
--buffer_compress_percentage=$i --filename=testfile.$i --eta=never
--output=/dev/null
done

for i in {0..100}; do
  gzip -v testfile.$i &>> gzip.txt
done

rm *.gz

Below are compression results for 4K and then 128K.  For 128K, you can
see compression stops matching at around 50%.

bs=4k

testfile.0:      0.3% -- replaced with testfile.0.gz
testfile.1:      1.2% -- replaced with testfile.1.gz
testfile.2:      2.2% -- replaced with testfile.2.gz
testfile.3:      3.2% -- replaced with testfile.3.gz
testfile.4:      4.1% -- replaced with testfile.4.gz
testfile.5:      5.1% -- replaced with testfile.5.gz
testfile.6:      6.1% -- replaced with testfile.6.gz
testfile.7:      7.1% -- replaced with testfile.7.gz
testfile.8:      8.1% -- replaced with testfile.8.gz
testfile.9:      9.1% -- replaced with testfile.9.gz
testfile.10:     10.1% -- replaced with testfile.10.gz
testfile.11:     11.1% -- replaced with testfile.11.gz
testfile.12:     12.1% -- replaced with testfile.12.gz
testfile.13:     13.1% -- replaced with testfile.13.gz
testfile.14:     14.1% -- replaced with testfile.14.gz
testfile.15:     15.1% -- replaced with testfile.15.gz
testfile.16:     16.0% -- replaced with testfile.16.gz
testfile.17:     17.0% -- replaced with testfile.17.gz
testfile.18:     18.0% -- replaced with testfile.18.gz
testfile.19:     19.0% -- replaced with testfile.19.gz
testfile.20:     20.0% -- replaced with testfile.20.gz
testfile.21:     21.0% -- replaced with testfile.21.gz
testfile.22:     22.0% -- replaced with testfile.22.gz
testfile.23:     23.0% -- replaced with testfile.23.gz
testfile.24:     24.0% -- replaced with testfile.24.gz
testfile.25:     25.0% -- replaced with testfile.25.gz
testfile.26:     25.9% -- replaced with testfile.26.gz
testfile.27:     26.9% -- replaced with testfile.27.gz
testfile.28:     27.9% -- replaced with testfile.28.gz
testfile.29:     28.9% -- replaced with testfile.29.gz
testfile.30:     29.9% -- replaced with testfile.30.gz
testfile.31:     30.9% -- replaced with testfile.31.gz
testfile.32:     31.9% -- replaced with testfile.32.gz
testfile.33:     32.9% -- replaced with testfile.33.gz
testfile.34:     33.9% -- replaced with testfile.34.gz
testfile.35:     34.9% -- replaced with testfile.35.gz
testfile.36:     35.9% -- replaced with testfile.36.gz
testfile.37:     36.9% -- replaced with testfile.37.gz
testfile.38:     37.9% -- replaced with testfile.38.gz
testfile.39:     38.9% -- replaced with testfile.39.gz
testfile.40:     39.9% -- replaced with testfile.40.gz
testfile.41:     40.8% -- replaced with testfile.41.gz
testfile.42:     41.8% -- replaced with testfile.42.gz
testfile.43:     42.8% -- replaced with testfile.43.gz
testfile.44:     43.8% -- replaced with testfile.44.gz
testfile.45:     44.8% -- replaced with testfile.45.gz
testfile.46:     45.8% -- replaced with testfile.46.gz
testfile.47:     46.8% -- replaced with testfile.47.gz
testfile.48:     47.8% -- replaced with testfile.48.gz
testfile.49:     48.8% -- replaced with testfile.49.gz
testfile.50:     49.8% -- replaced with testfile.50.gz
testfile.51:     50.3% -- replaced with testfile.51.gz
testfile.52:     51.2% -- replaced with testfile.52.gz
testfile.53:     52.3% -- replaced with testfile.53.gz
testfile.54:     53.2% -- replaced with testfile.54.gz
testfile.55:     54.2% -- replaced with testfile.55.gz
testfile.56:     55.3% -- replaced with testfile.56.gz
testfile.57:     56.1% -- replaced with testfile.57.gz
testfile.58:     57.2% -- replaced with testfile.58.gz
testfile.59:     58.2% -- replaced with testfile.59.gz
testfile.60:     59.2% -- replaced with testfile.60.gz
testfile.61:     60.2% -- replaced with testfile.61.gz
testfile.62:     61.2% -- replaced with testfile.62.gz
testfile.63:     62.2% -- replaced with testfile.63.gz
testfile.64:     63.2% -- replaced with testfile.64.gz
testfile.65:     64.1% -- replaced with testfile.65.gz
testfile.66:     65.2% -- replaced with testfile.66.gz
testfile.67:     66.2% -- replaced with testfile.67.gz
testfile.68:     67.2% -- replaced with testfile.68.gz
testfile.69:     68.2% -- replaced with testfile.69.gz
testfile.70:     69.2% -- replaced with testfile.70.gz
testfile.71:     70.2% -- replaced with testfile.71.gz
testfile.72:     71.2% -- replaced with testfile.72.gz
testfile.73:     72.2% -- replaced with testfile.73.gz
testfile.74:     73.2% -- replaced with testfile.74.gz
testfile.75:     74.3% -- replaced with testfile.75.gz
testfile.76:     75.2% -- replaced with testfile.76.gz
testfile.77:     76.1% -- replaced with testfile.77.gz
testfile.78:     77.2% -- replaced with testfile.78.gz
testfile.79:     78.2% -- replaced with testfile.79.gz
testfile.80:     79.1% -- replaced with testfile.80.gz
testfile.81:     80.1% -- replaced with testfile.81.gz
testfile.82:     81.2% -- replaced with testfile.82.gz
testfile.83:     82.2% -- replaced with testfile.83.gz
testfile.84:     83.1% -- replaced with testfile.84.gz
testfile.85:     84.2% -- replaced with testfile.85.gz
testfile.86:     85.1% -- replaced with testfile.86.gz
testfile.87:     86.1% -- replaced with testfile.87.gz
testfile.88:     87.2% -- replaced with testfile.88.gz
testfile.89:     88.2% -- replaced with testfile.89.gz
testfile.90:     89.2% -- replaced with testfile.90.gz
testfile.91:     90.1% -- replaced with testfile.91.gz
testfile.92:     91.1% -- replaced with testfile.92.gz
testfile.93:     92.0% -- replaced with testfile.93.gz
testfile.94:     93.2% -- replaced with testfile.94.gz
testfile.95:     94.1% -- replaced with testfile.95.gz
testfile.96:     95.0% -- replaced with testfile.96.gz
testfile.97:     96.4% -- replaced with testfile.97.gz
testfile.98:     97.3% -- replaced with testfile.98.gz
testfile.99:     98.5% -- replaced with testfile.99.gz
testfile.100:     99.9% -- replaced with testfile.100.gz

bs=128k

testfile.0:      0.3% -- replaced with testfile.0.gz
testfile.1:      1.3% -- replaced with testfile.1.gz
testfile.2:      2.3% -- replaced with testfile.2.gz
testfile.3:      3.3% -- replaced with testfile.3.gz
testfile.4:      4.2% -- replaced with testfile.4.gz
testfile.5:      5.2% -- replaced with testfile.5.gz
testfile.6:      6.2% -- replaced with testfile.6.gz
testfile.7:      7.2% -- replaced with testfile.7.gz
testfile.8:      8.2% -- replaced with testfile.8.gz
testfile.9:      9.2% -- replaced with testfile.9.gz
testfile.10:     10.2% -- replaced with testfile.10.gz
testfile.11:     11.2% -- replaced with testfile.11.gz
testfile.12:     12.2% -- replaced with testfile.12.gz
testfile.13:     13.2% -- replaced with testfile.13.gz
testfile.14:     14.2% -- replaced with testfile.14.gz
testfile.15:     15.2% -- replaced with testfile.15.gz
testfile.16:     16.1% -- replaced with testfile.16.gz
testfile.17:     17.1% -- replaced with testfile.17.gz
testfile.18:     18.1% -- replaced with testfile.18.gz
testfile.19:     19.1% -- replaced with testfile.19.gz
testfile.20:     20.1% -- replaced with testfile.20.gz
testfile.21:     21.1% -- replaced with testfile.21.gz
testfile.22:     22.1% -- replaced with testfile.22.gz
testfile.23:     23.1% -- replaced with testfile.23.gz
testfile.24:     24.1% -- replaced with testfile.24.gz
testfile.25:     25.1% -- replaced with testfile.25.gz
testfile.26:     26.1% -- replaced with testfile.26.gz
testfile.27:     27.1% -- replaced with testfile.27.gz
testfile.28:     28.1% -- replaced with testfile.28.gz
testfile.29:     29.1% -- replaced with testfile.29.gz
testfile.30:     30.0% -- replaced with testfile.30.gz
testfile.31:     31.0% -- replaced with testfile.31.gz
testfile.32:     32.0% -- replaced with testfile.32.gz
testfile.33:     33.0% -- replaced with testfile.33.gz
testfile.34:     34.0% -- replaced with testfile.34.gz
testfile.35:     35.0% -- replaced with testfile.35.gz
testfile.36:     36.0% -- replaced with testfile.36.gz
testfile.37:     37.0% -- replaced with testfile.37.gz
testfile.38:     38.0% -- replaced with testfile.38.gz
testfile.39:     39.0% -- replaced with testfile.39.gz
testfile.40:     40.0% -- replaced with testfile.40.gz
testfile.41:     41.0% -- replaced with testfile.41.gz
testfile.42:     42.0% -- replaced with testfile.42.gz
testfile.43:     43.0% -- replaced with testfile.43.gz
testfile.44:     44.0% -- replaced with testfile.44.gz
testfile.45:     45.0% -- replaced with testfile.45.gz
testfile.46:     46.0% -- replaced with testfile.46.gz
testfile.47:     47.0% -- replaced with testfile.47.gz
testfile.48:     47.9% -- replaced with testfile.48.gz
testfile.49:     48.9% -- replaced with testfile.49.gz
testfile.50:     49.9% -- replaced with testfile.50.gz
testfile.51:     48.9% -- replaced with testfile.51.gz
testfile.52:     47.9% -- replaced with testfile.52.gz
testfile.53:     47.0% -- replaced with testfile.53.gz
testfile.54:     46.0% -- replaced with testfile.54.gz
testfile.55:     45.0% -- replaced with testfile.55.gz
testfile.56:     44.0% -- replaced with testfile.56.gz
testfile.57:     43.0% -- replaced with testfile.57.gz
testfile.58:     42.0% -- replaced with testfile.58.gz
testfile.59:     41.0% -- replaced with testfile.59.gz
testfile.60:     40.0% -- replaced with testfile.60.gz
testfile.61:     39.0% -- replaced with testfile.61.gz
testfile.62:     38.0% -- replaced with testfile.62.gz
testfile.63:     37.0% -- replaced with testfile.63.gz
testfile.64:     36.0% -- replaced with testfile.64.gz
testfile.65:     35.0% -- replaced with testfile.65.gz
testfile.66:     34.0% -- replaced with testfile.66.gz
testfile.67:     34.0% -- replaced with testfile.67.gz
testfile.68:     36.0% -- replaced with testfile.68.gz
testfile.69:     38.0% -- replaced with testfile.69.gz
testfile.70:     40.0% -- replaced with testfile.70.gz
testfile.71:     42.0% -- replaced with testfile.71.gz
testfile.72:     43.9% -- replaced with testfile.72.gz
testfile.73:     45.9% -- replaced with testfile.73.gz
testfile.74:     47.9% -- replaced with testfile.74.gz
testfile.75:     49.9% -- replaced with testfile.75.gz
testfile.76:     47.9% -- replaced with testfile.76.gz
testfile.77:     45.9% -- replaced with testfile.77.gz
testfile.78:     43.9% -- replaced with testfile.78.gz
testfile.79:     41.9% -- replaced with testfile.79.gz
testfile.80:     40.0% -- replaced with testfile.80.gz
testfile.81:     42.9% -- replaced with testfile.81.gz
testfile.82:     45.9% -- replaced with testfile.82.gz
testfile.83:     48.9% -- replaced with testfile.83.gz
testfile.84:     47.9% -- replaced with testfile.84.gz
testfile.85:     44.9% -- replaced with testfile.85.gz
testfile.86:     43.9% -- replaced with testfile.86.gz
testfile.87:     47.9% -- replaced with testfile.87.gz
testfile.88:     87.4% -- replaced with testfile.88.gz
testfile.89:     88.4% -- replaced with testfile.89.gz
testfile.90:     89.4% -- replaced with testfile.90.gz
testfile.91:     90.4% -- replaced with testfile.91.gz
testfile.92:     91.4% -- replaced with testfile.92.gz
testfile.93:     92.4% -- replaced with testfile.93.gz
testfile.94:     93.4% -- replaced with testfile.94.gz
testfile.95:     94.4% -- replaced with testfile.95.gz
testfile.96:     95.4% -- replaced with testfile.96.gz
testfile.97:     96.4% -- replaced with testfile.97.gz
testfile.98:     97.4% -- replaced with testfile.98.gz
testfile.99:     98.4% -- replaced with testfile.99.gz
testfile.100:     99.9% -- replaced with testfile.100.gz

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

On Fri, Nov 20, 2015 at 9:29 AM, Matthew Eaton <m.eaton82@gmail.com> wrote:
> I noticed that sometimes using --buffer_compress_percentage would not
> result in the desired compression.  After running some tests I found
> that only some block sizes are affected.

Forgot to mention my fio version is: fio-2.2.11-17-gd883ef

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/20/2015 10:29 AM, Matthew Eaton wrote:
> I noticed that sometimes using --buffer_compress_percentage would not
> result in the desired compression.  After running some tests I found
> that only some block sizes are affected.
>
> 512b, 4K, 8K, 16K, and 32K seem to work as expected.
>
> 64K, 128K, 256K, 512K, and 1024K exhibited the bug.  I have not tested
> beyond 1024K.
>
> Here is the test script I used only changing block size for each test.
>
> for i in {0..100}; do
>    fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
> --iodepth=32 --size=512m --refill_buffers
> --buffer_compress_percentage=$i --filename=testfile.$i --eta=never
> --output=/dev/null
> done
>
> for i in {0..100}; do
>    gzip -v testfile.$i &>> gzip.txt
> done
>
> rm *.gz
>
> Below are compression results for 4K and then 128K.  For 128K, you can
> see compression stops matching at around 50%.

I think this is an artifact of how gzip compression works, it doesn't 
have an unlimited size window. It's 32k, iirc. Fio should fill so that 
it ideally would compress to the given size with an ideal compression 
algorithm.

You'd probably want to use buffer_compress_chunk=x to always force fio 
to operate in sizes of that for compression, if you want to ensure that 
gzip would be able to compress to the specified ratio. I do think we 
have a loop missing though for that to do what you need, let me test 
that and report back.

-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1786 bytes --]

On 11/23/2015 09:08 AM, Jens Axboe wrote:
> On 11/20/2015 10:29 AM, Matthew Eaton wrote:
>> I noticed that sometimes using --buffer_compress_percentage would not
>> result in the desired compression.  After running some tests I found
>> that only some block sizes are affected.
>>
>> 512b, 4K, 8K, 16K, and 32K seem to work as expected.
>>
>> 64K, 128K, 256K, 512K, and 1024K exhibited the bug.  I have not tested
>> beyond 1024K.
>>
>> Here is the test script I used only changing block size for each test.
>>
>> for i in {0..100}; do
>>    fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
>> --iodepth=32 --size=512m --refill_buffers
>> --buffer_compress_percentage=$i --filename=testfile.$i --eta=never
>> --output=/dev/null
>> done
>>
>> for i in {0..100}; do
>>    gzip -v testfile.$i &>> gzip.txt
>> done
>>
>> rm *.gz
>>
>> Below are compression results for 4K and then 128K.  For 128K, you can
>> see compression stops matching at around 50%.
>
> I think this is an artifact of how gzip compression works, it doesn't
> have an unlimited size window. It's 32k, iirc. Fio should fill so that
> it ideally would compress to the given size with an ideal compression
> algorithm.
>
> You'd probably want to use buffer_compress_chunk=x to always force fio
> to operate in sizes of that for compression, if you want to ensure that
> gzip would be able to compress to the specified ratio. I do think we
> have a loop missing though for that to do what you need, let me test
> that and report back.

I think things should work for all sizes, if you add this patch and then 
add buffer_compress_chunk=32k or similar to your job file. You can do 
that for smaller block sizes as well, fio will use the minimum of that 
chunk size or the write block size.

-- 
Jens Axboe


[-- Attachment #2: big-chunk-compress.patch --]
[-- Type: text/x-patch, Size: 1303 bytes --]

diff --git a/io_u.c b/io_u.c
index 6b6b47d0a1e1..b1aa4141cdbc 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1919,6 +1919,7 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 		unsigned int perc = td->o.compress_percentage;
 		struct frand_state *rs;
 		unsigned int left = max_bs;
+		unsigned int this_write;
 
 		do {
 			rs = get_buf_state(td);
@@ -1926,20 +1927,21 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 			min_write = min(min_write, left);
 
 			if (perc) {
-				unsigned int seg = min_write;
-
-				seg = min(min_write, td->o.compress_chunk);
-				if (!seg)
-					seg = min_write;
-
-				fill_random_buf_percentage(rs, buf, perc, seg,
-					min_write, o->buffer_pattern,
-						   o->buffer_pattern_bytes);
-			} else
+				this_write = min(min_write, td->o.compress_chunk);
+				if (!this_write)
+					this_write = min_write;
+
+				fill_random_buf_percentage(rs, buf, perc,
+					this_write, this_write,
+					o->buffer_pattern,
+					o->buffer_pattern_bytes);
+			} else {
 				fill_random_buf(rs, buf, min_write);
+				this_write = min_write;
+			}
 
-			buf += min_write;
-			left -= min_write;
+			buf += this_write;
+			left -= this_write;
 			save_buf_state(td, rs);
 		} while (left);
 	} else if (o->buffer_pattern_bytes)

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

>> I think this is an artifact of how gzip compression works, it doesn't
>> have an unlimited size window. It's 32k, iirc. Fio should fill so that
>> it ideally would compress to the given size with an ideal compression
>> algorithm.
>>
>> You'd probably want to use buffer_compress_chunk=x to always force fio
>> to operate in sizes of that for compression, if you want to ensure that
>> gzip would be able to compress to the specified ratio. I do think we
>> have a loop missing though for that to do what you need, let me test
>> that and report back.
>
>
> I think things should work for all sizes, if you add this patch and then add
> buffer_compress_chunk=32k or similar to your job file. You can do that for
> smaller block sizes as well, fio will use the minimum of that chunk size or
> the write block size.
>
> --
> Jens Axboe
>

Applied the patch and added buffer_compress_chunk=32k and now it's
working for all block sizes. Thank you for the info and the patch!

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/23/2015 03:15 PM, Matthew Eaton wrote:
>>> I think this is an artifact of how gzip compression works, it doesn't
>>> have an unlimited size window. It's 32k, iirc. Fio should fill so that
>>> it ideally would compress to the given size with an ideal compression
>>> algorithm.
>>>
>>> You'd probably want to use buffer_compress_chunk=x to always force fio
>>> to operate in sizes of that for compression, if you want to ensure that
>>> gzip would be able to compress to the specified ratio. I do think we
>>> have a loop missing though for that to do what you need, let me test
>>> that and report back.
>>
>>
>> I think things should work for all sizes, if you add this patch and then add
>> buffer_compress_chunk=32k or similar to your job file. You can do that for
>> smaller block sizes as well, fio will use the minimum of that chunk size or
>> the write block size.
>>
>> --
>> Jens Axboe
>>
>
> Applied the patch and added buffer_compress_chunk=32k and now it's
> working for all block sizes. Thank you for the info and the patch!

Perfect, thanks for reporting and followup testing.

-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

> Perfect, thanks for reporting and followup testing.
>
> --
> Jens Axboe

Unfortunately I am having an issue with buffer_compress_chunk in the
2.2.12 release.  Removing buffer_compress_chunk from the tests below
fixes the issues.

fio --name=test --rw=write --bs=4k --ioengine=libaio --direct=1
--iodepth=32 --size=1g --refill_buffers
--buffer_compress_percentage=50 --buffer_compress_chunk=32k
--filename=testfile
test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.2.12
Starting 1 process
fio: pid=24516, got signal=11


Run status group 0 (all jobs):

Disk stats (read/write):
  sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
fio: file hash not empty on exit


fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
--iodepth=32 --size=1g --refill_buffers
--buffer_compress_percentage=50 --buffer_compress_chunk=32k
--filename=testfile
test: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K,
ioengine=libaio, iodepth=32
fio-2.2.12
Starting 1 process
Segmentation fault (core dumped)

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/25/2015 10:16 AM, Matthew Eaton wrote:
>> Perfect, thanks for reporting and followup testing.
>>
>> --
>> Jens Axboe
>
> Unfortunately I am having an issue with buffer_compress_chunk in the
> 2.2.12 release.  Removing buffer_compress_chunk from the tests below
> fixes the issues.
>
> fio --name=test --rw=write --bs=4k --ioengine=libaio --direct=1
> --iodepth=32 --size=1g --refill_buffers
> --buffer_compress_percentage=50 --buffer_compress_chunk=32k
> --filename=testfile
> test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
> fio-2.2.12
> Starting 1 process
> fio: pid=24516, got signal=11
>
>
> Run status group 0 (all jobs):
>
> Disk stats (read/write):
>    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
> fio: file hash not empty on exit
>
>
> fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
> --iodepth=32 --size=1g --refill_buffers
> --buffer_compress_percentage=50 --buffer_compress_chunk=32k
> --filename=testfile
> test: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K,
> ioengine=libaio, iodepth=32
> fio-2.2.12
> Starting 1 process
> Segmentation fault (core dumped)

That's odd, I can't reproduce that. Could this just be a bad build? Fio 
sometimes fails to rebuild correctly between version. Does it persist 
across a make clean && make?


-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

> That's odd, I can't reproduce that. Could this just be a bad build? Fio
> sometimes fails to rebuild correctly between version. Does it persist across
> a make clean && make?
>
>
> --
> Jens Axboe
>

Weird!  I will look into this again on Monday when I am back at work.

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/25/2015 05:55 PM, Matthew Eaton wrote:
>> That's odd, I can't reproduce that. Could this just be a bad build? Fio
>> sometimes fails to rebuild correctly between version. Does it persist across
>> a make clean && make?
>>
>>
>> --
>> Jens Axboe
>>
>
> Weird!  I will look into this again on Monday when I am back at work.

I think I found and fixed the main issue why the rebuild issue would 
happen, init.c didn't get the right dependencies. Hopefully a make clean 
won't be needed going forward.

I still think that is your issue, I don't see any problems with the 
buffer filling. So do let me know when you test on Monday, thanks.

-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

> I think I found and fixed the main issue why the rebuild issue would happen,
> init.c didn't get the right dependencies. Hopefully a make clean won't be
> needed going forward.
>
> I still think that is your issue, I don't see any problems with the buffer
> filling. So do let me know when you test on Monday, thanks.
>
> --
> Jens Axboe
>

I tested again trying a make clean as well as a fresh clone from git
but I could still reproduce.  This system is Ubuntu 14.04, so for
kicks I moved to a completely different system and installed Fedora 23
and fio from git, but had the same problem!

This is my configure output.  Could I be missing something?

Operating system              Linux
CPU                           x86_64
Big endian                    no
Compiler                      gcc
Cross compile                 no

Static build                  no
Wordsize                      64
zlib                          yes
Linux AIO support             yes
POSIX AIO support             yes
POSIX AIO support needs -lrt  yes
POSIX AIO fsync               yes
Solaris AIO support           no
__sync_fetch_and_add          yes
libverbs                      yes
rdmacm                        yes
Linux fallocate               yes
POSIX fadvise                 yes
POSIX fallocate               yes
sched_setaffinity(3 arg)      yes
sched_setaffinity(2 arg)      no
clock_gettime                 yes
CLOCK_MONOTONIC               yes
CLOCK_MONOTONIC_RAW           yes
CLOCK_MONOTONIC_PRECISE       no
gettimeofday                  yes
fdatasync                     yes
sync_file_range               yes
EXT4 move extent              yes
Linux splice(2)               yes
GUASI                         no
Fusion-io atomic engine       no
libnuma                       no
strsep                        yes
strcasestr                    yes
strlcat                       no
getopt_long_only()            yes
inet_aton                     yes
socklen_t                     yes
__thread                      yes
RUSAGE_THREAD                 yes
SCHED_IDLE                    yes
TCP_NODELAY                   yes
Net engine window_size        yes
TCP_MAXSEG                    yes
RLIMIT_MEMLOCK                yes
pwritev/preadv                yes
IPv6 helpers                  yes
Rados Block Device engine     no
setvbuf                       yes
Gluster API engine            no
s390_z196_facilities          no
HDFS engine                   no
MTD                           yes
lex/yacc for arithmetic       no
getmntent                     yes
getmntinfo                    no
Static Assert                 yes

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/30/2015 02:17 PM, Matthew Eaton wrote:
>> I think I found and fixed the main issue why the rebuild issue would happen,
>> init.c didn't get the right dependencies. Hopefully a make clean won't be
>> needed going forward.
>>
>> I still think that is your issue, I don't see any problems with the buffer
>> filling. So do let me know when you test on Monday, thanks.
>>
>> --
>> Jens Axboe
>>
>
> I tested again trying a make clean as well as a fresh clone from git
> but I could still reproduce.  This system is Ubuntu 14.04, so for
> kicks I moved to a completely different system and installed Fedora 23
> and fio from git, but had the same problem!
>
> This is my configure output.  Could I be missing something?

Nope, that part should not matter. I'm probably wrong, then. I'll try 
and reproduce!

BTW, the rebuild bug should hopefully be fixed, so I can trust reports 
like this more in the future.


-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Jens Axboe]

On 11/30/2015 02:20 PM, Jens Axboe wrote:
> On 11/30/2015 02:17 PM, Matthew Eaton wrote:
>>> I think I found and fixed the main issue why the rebuild issue would
>>> happen,
>>> init.c didn't get the right dependencies. Hopefully a make clean
>>> won't be
>>> needed going forward.
>>>
>>> I still think that is your issue, I don't see any problems with the
>>> buffer
>>> filling. So do let me know when you test on Monday, thanks.
>>>
>>> --
>>> Jens Axboe
>>>
>>
>> I tested again trying a make clean as well as a fresh clone from git
>> but I could still reproduce.  This system is Ubuntu 14.04, so for
>> kicks I moved to a completely different system and installed Fedora 23
>> and fio from git, but had the same problem!
>>
>> This is my configure output.  Could I be missing something?
>
> Nope, that part should not matter. I'm probably wrong, then. I'll try
> and reproduce!

Try current -git, I think I found it.

-- 
Jens Axboe

Re: Bug with --buffer_compress_percentage

[Matthew Eaton]

> Try current -git, I think I found it.
>
> --
> Jens Axboe
>

That did the trick!  Thanks again!