blkio weights not honoured with lvm setup and on amazon ebs

blkio weights not honoured with lvm setup and on amazon ebs

[Himadri Sarkar]

Hi,

I was doing some experiments to test out blkio with the following
setups and was unable to get the expected behavior. It will be great
if someone can help me understand what might have gone wrong with my
setup.

Setup 1
hardware: d2.xlarge machine on aws (It has 3 * 2 TB hdd)
logical volume: setup an lvm to use 2 disks out of 3
file system: xfs
IO Scheduler: cfq
blkio weights: test1 has weight 1000, test2 has weight 500

Now when I run the tests as given in
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
using dd (I only executed read tests on already written files)
I found that both the processes were getting equal time share
(blkio.time) and serviced bytes (blkio.io_service_bytes) irrespective
of differential weights.

Now when I modified the setup by not having lvm setup and just
creating a 2TB file system out of a single hdd it worked i.e. serviced
bytes were in the ratio 2 : 1

On the other hand when I tested read with fio

specifically

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
--name=test2 --filename=file2 --bs=4k --iodepth=64 --size=4G
--readwrite=randrw --rwmixread=100

Then also it worked even with the lvm setup. Which makes be believe
that with Setup 1 it is not working due to buffered reads. (But then
the same thing also worked without lvm)

Setup 2
hardware: c3.xlarge machine on aws with 300GB EBS (which I believe are
like SAN storage)
logical volume: setup an lvm
file system: xfs
IO Scheduler: cfq
blkio weights: test1 has weight 1000, test 2 has weight 500

Now with this setup I was not able to get it working in any of the
cases i.e. with lvm, without lvm, with fio sync reads.
Is it the problem with using EBS volumes? Is there way in which I can
get blkio to work with Setup 1 and Setup 2? Please let me know if
providing more data points might help.


Regards
- Himadri

Re: blkio weights not honoured with lvm setup and on amazon ebs

[nborisov]

On 25.04.2016 20:53, Himadri Sarkar wrote:
> Hi,
> 
> I was doing some experiments to test out blkio with the following
> setups and was unable to get the expected behavior. It will be great
> if someone can help me understand what might have gone wrong with my
> setup.
> 
> Setup 1
> hardware: d2.xlarge machine on aws (It has 3 * 2 TB hdd)
> logical volume: setup an lvm to use 2 disks out of 3
> file system: xfs
> IO Scheduler: cfq
> blkio weights: test1 has weight 1000, test2 has weight 500
> 
> Now when I run the tests as given in
> https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
> using dd (I only executed read tests on already written files)
> I found that both the processes were getting equal time share
> (blkio.time) and serviced bytes (blkio.io_service_bytes) irrespective
> of differential weights.
> 
> Now when I modified the setup by not having lvm setup and just
> creating a 2TB file system out of a single hdd it worked i.e. serviced
> bytes were in the ratio 2 : 1
> 
> On the other hand when I tested read with fio
> 
> specifically
> 
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test2 --filename=file2 --bs=4k --iodepth=64 --size=4G
> --readwrite=randrw --rwmixread=100
> 
> Then also it worked even with the lvm setup. Which makes be believe
> that with Setup 1 it is not working due to buffered reads. (But then
> the same thing also worked without lvm)

This is a well-known limitation to LVM, which supposedly was fixed in a
recent kernel. For more information check this email thread:
https://www.redhat.com/archives/dm-devel/2016-February/msg00183.html

Then a patch was proposed in the following thread:
https://www.redhat.com/archives/dm-devel/2016-March/msg00006.html

This was tested by me and the proportional-based limits were working,
provided that the devices hosting the LVM VG were using CFQ as their io
scheduler.

Regards,
Nikolay

Re: blkio weights not honoured with lvm setup and on amazon ebs

[Himadri Sarkar]

Thanks for your reply.
Does it also not work with Amazon EBS and SAN based storage?
Regards
- Himadri


On Tue, Apr 26, 2016 at 12:01 AM, nborisov <n.borisov.lkml-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
> On 25.04.2016 20:53, Himadri Sarkar wrote:
>> Hi,
>>
>> I was doing some experiments to test out blkio with the following
>> setups and was unable to get the expected behavior. It will be great
>> if someone can help me understand what might have gone wrong with my
>> setup.
>>
>> Setup 1
>> hardware: d2.xlarge machine on aws (It has 3 * 2 TB hdd)
>> logical volume: setup an lvm to use 2 disks out of 3
>> file system: xfs
>> IO Scheduler: cfq
>> blkio weights: test1 has weight 1000, test2 has weight 500
>>
>> Now when I run the tests as given in
>> https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
>> using dd (I only executed read tests on already written files)
>> I found that both the processes were getting equal time share
>> (blkio.time) and serviced bytes (blkio.io_service_bytes) irrespective
>> of differential weights.
>>
>> Now when I modified the setup by not having lvm setup and just
>> creating a 2TB file system out of a single hdd it worked i.e. serviced
>> bytes were in the ratio 2 : 1
>>
>> On the other hand when I tested read with fio
>>
>> specifically
>>
>> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
>> --name=test2 --filename=file2 --bs=4k --iodepth=64 --size=4G
>> --readwrite=randrw --rwmixread=100
>>
>> Then also it worked even with the lvm setup. Which makes be believe
>> that with Setup 1 it is not working due to buffered reads. (But then
>> the same thing also worked without lvm)
>
> This is a well-known limitation to LVM, which supposedly was fixed in a
> recent kernel. For more information check this email thread:
> https://www.redhat.com/archives/dm-devel/2016-February/msg00183.html
>
> Then a patch was proposed in the following thread:
> https://www.redhat.com/archives/dm-devel/2016-March/msg00006.html
>
> This was tested by me and the proportional-based limits were working,
> provided that the devices hosting the LVM VG were using CFQ as their io
> scheduler.
>
> Regards,
> Nikolay