fio memory allocation problems

fio memory allocation problems

[Jiri Horky]

[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]

Hi,

I think I found a bug. If the device (file) used in fio has "wrong" 
size, and is used with "wrong" block size, the fio fails to allocate 
memory for randommap.
Following conditions must be true, considering just one thread for 
simplicity:

  1) number of blocks on the device are too big to fill in the default's 
pool
  2) number of blocks on the device must be "right", so the size of 
newly allocated pool by smalloc.c:add_pool is exactly as big as 
requested size (there are some roundings in place, so not every size 
triggers this error)

When this is true, function smalloc.c:__smalloc_pool fails to use the 
newly created (exactly big enough) pool, because of this line:

                 idx = find_next_zero(pool->bitmap[i], last_idx);

which always returns non-zero value (the minimum returned value is 
last_idx +1), and so in our case the idx is 1 even though, the block 
number 0 is free. As the direct consequence, the function doesn't find 
enough free space in the given pool and returns NULL pointer. This leads 
to another try to allocate enough memory in smalloc.c:smalloc_real 
function, and so on...

The behavior could be reprocuded using both stable 1.57 and git version 
of the tool. The values that triggers the error are:
init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, 
blocks: 5371092672, num_maps: 83923323, alloc size: 671386584

whereas this block device is fine:
init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, 
blocks: 4882811520, num_maps: 76293930, alloc size: 610351440

the configuration file used:

[global]
description=CESNET_test
[cesnet]
filename=/dev/sdd
rw=randread
size=10000g
ioengine=libaio
iodepth=4
bs=4k
runtime=10m
time_based
numjobs=1
#norandommap
group_reporting


I am not sure I fully understand all the logic in smalloc.c so I am 
afraid I would break something with a trivial patch.

Let me know, if you need more information

Best Regards
Jiri Horky






[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6917 bytes --]

Re: fio memory allocation problems

[Jens Axboe]

On 2011-08-12 16:42, Jiri Horky wrote:
> Hi,
> 
> I think I found a bug. If the device (file) used in fio has "wrong" size, and is used with "wrong" block size, the fio fails to allocate memory for randommap.
> Following conditions must be true, considering just one thread for simplicity:
> 
>  1) number of blocks on the device are too big to fill in the default's pool
>  2) number of blocks on the device must be "right", so the size of newly allocated pool by smalloc.c:add_pool is exactly as big as requested size (there are some roundings in place, so not every size triggers this error)
> 
> When this is true, function smalloc.c:__smalloc_pool fails to use the newly created (exactly big enough) pool, because of this line:
> 
>                 idx = find_next_zero(pool->bitmap[i], last_idx);
> 
> which always returns non-zero value (the minimum returned value is last_idx +1), and so in our case the idx is 1 even though, the block number 0 is free. As the direct consequence, the function doesn't find enough free space in the given pool and returns NULL pointer. This leads to another try to allocate enough memory in smalloc.c:smalloc_real function, and so on...
> 
> The behavior could be reprocuded using both stable 1.57 and git version of the tool. The values that triggers the error are:
> init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, blocks: 5371092672, num_maps: 83923323, alloc size: 671386584
> 
> whereas this block device is fine:
> init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, blocks: 4882811520, num_maps: 76293930, alloc size: 610351440
> 
> the configuration file used:
> 
> [global]
> description=CESNET_test
> [cesnet]
> filename=/dev/sdd
> rw=randread
> size=10000g
> ioengine=libaio
> iodepth=4
> bs=4k
> runtime=10m
> time_based
> numjobs=1
> #norandommap
> group_reporting
> 
> 
> I am not sure I fully understand all the logic in smalloc.c so I am afraid I would break something with a trivial patch.
> 
> Let me know, if you need more information

That does indeed look like a bug in smalloc, bummer. I'll try and take a
look at it, if you do have a test patch you are using, please send it
along.

-- 
Jens Axboe

Re: fio memory allocation problems

[Jiri Horky]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

Hi Jens,

the patch is attached. It resolves the issue but I would not bet it 
works in all scenarios.

Cheers
Jiri
------------

--- fio-head/smalloc.c  2011-08-15 10:40:24.000000000 +0200
+++ fio-head-patched/smalloc.c  2011-08-15 10:49:32.000000000 +0200
@@ -168,11 +168,11 @@
         blocks_iter(pool, pool_idx, idx, nr_blocks, mask_clear);
  }

-static int find_next_zero(int word, int start)
+static int find_zero(int word, int start)
  {
         assert(word != -1U);
-       word >>= (start + 1);
-       return ffz(word) + start + 1;
+       word >>= (start);
+       return ffz(word) + start;
  }

  static int add_pool(struct pool *pool, unsigned int alloc_size)
@@ -374,7 +374,7 @@
                         continue;
                 }

-               idx = find_next_zero(pool->bitmap[i], last_idx);
+               idx = find_zero(pool->bitmap[i], last_idx);
                 if (!blocks_free(pool, i, idx, nr_blocks)) {
                         idx += nr_blocks;
                         if (idx < SMALLOC_BPI)


On 08/15/2011 10:29 AM, Jens Axboe wrote:
> On 2011-08-12 16:42, Jiri Horky wrote:
>> Hi,
>>
>> I think I found a bug. If the device (file) used in fio has "wrong" size, and is used with "wrong" block size, the fio fails to allocate memory for randommap.
>> Following conditions must be true, considering just one thread for simplicity:
>>
>>   1) number of blocks on the device are too big to fill in the default's pool
>>   2) number of blocks on the device must be "right", so the size of newly allocated pool by smalloc.c:add_pool is exactly as big as requested size (there are some roundings in place, so not every size triggers this error)
>>
>> When this is true, function smalloc.c:__smalloc_pool fails to use the newly created (exactly big enough) pool, because of this line:
>>
>>                  idx = find_next_zero(pool->bitmap[i], last_idx);
>>
>> which always returns non-zero value (the minimum returned value is last_idx +1), and so in our case the idx is 1 even though, the block number 0 is free. As the direct consequence, the function doesn't find enough free space in the given pool and returns NULL pointer. This leads to another try to allocate enough memory in smalloc.c:smalloc_real function, and so on...
>>
>> The behavior could be reprocuded using both stable 1.57 and git version of the tool. The values that triggers the error are:
>> init_random_map - real_file_size: 21999995584512, o.rw_min_bs: 4096, blocks: 5371092672, num_maps: 83923323, alloc size: 671386584
>>
>> whereas this block device is fine:
>> init_random_map - real_file_size: 19999995985920, o.rw_min_bs: 4096, blocks: 4882811520, num_maps: 76293930, alloc size: 610351440
>>
>> the configuration file used:
>>
>> [global]
>> description=CESNET_test
>> [cesnet]
>> filename=/dev/sdd
>> rw=randread
>> size=10000g
>> ioengine=libaio
>> iodepth=4
>> bs=4k
>> runtime=10m
>> time_based
>> numjobs=1
>> #norandommap
>> group_reporting
>>
>>
>> I am not sure I fully understand all the logic in smalloc.c so I am afraid I would break something with a trivial patch.
>>
>> Let me know, if you need more information
> That does indeed look like a bug in smalloc, bummer. I'll try and take a
> look at it, if you do have a test patch you are using, please send it
> along.
>


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6917 bytes --]

Re: fio memory allocation problems

[Jens Axboe]

On 2011-08-15 10:57, Jiri Horky wrote:
> Hi Jens,
> 
> the patch is attached. It resolves the issue but I would not bet it
> works in all scenarios.

I think it's OK. Just to be on the safe side, I ran it through my stest
app that I had saved from when I did smalloc, and it passes. I think the
+1 dates back to when the pools had a header.

-- 
Jens Axboe