From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Snitzer Subject: Re: how to set higher then 128 nr_requests on LV's Date: Mon, 14 May 2018 13:53:54 -0400 Message-ID: <20180514175354.GA15053@redhat.com> References: <20180514161312.GA14685@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20180514161312.GA14685@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Pankaj Agarwal Cc: Hannes Reinecke , dm-devel@redhat.com, linux-lvm@redhat.com List-Id: dm-devel.ids T24gTW9uLCBNYXkgMTQgMjAxOCBhdCAxMjoxM3BtIC0wNDAwLApNaWtlIFNuaXR6ZXIgPHNuaXR6 ZXJAcmVkaGF0LmNvbT4gd3JvdGU6Cgo+IE9uIFN1biwgTWF5IDEzIDIwMTggYXQgIDg6NDdwbSAt MDQwMCwKPiBQYW5rYWogQWdhcndhbCA8cGFua2FqQHJlbGVhc2VtYW5hZ2VyLmluPiB3cm90ZToK PiAKPiA+ICAgIEhpLAo+ID4gICAgSG93IGRvIGkgc2V0IHRoZSBucl9yZXF1ZXN0IHZhbHVlIGZv ciBMVidzIGFzIGl0J3Mgbm90IHdyaXRhYmxlIGxpa2Ugb3RoZXIKPiA+ICAgIGRyaXZlcyBvbiBh IGxpbnV4IHN5c3RlbS4KPiA+ICAgIExWJ3MgYXJlIHNldCBhcyBkbS0wIGFuZCBkbS0xIG9uIG15 IHN5c3RlbS4KPiA+ICAgICAjY2F0IC9zeXMvYmxvY2svZG0tMC9xdWV1ZS9ucl9yZXF1ZXN0cwo+ ID4gICAgMTI4Cj4gPiAgICAjIGVjaG8gMjU2ID4gL3N5cy9ibG9jay9kbS0wL3F1ZXVlL25yX3Jl cXVlc3RzCj4gPiAgICAtYmFzaDogZWNobzogd3JpdGUgZXJyb3I6IEludmFsaWQgYXJndW1lbnQK PiAKPiBJbiB0aGUgYmxvY2sgbGF5ZXIsIGJpby1iYXNlZCBkZXZpY2VzIGNhbm5vdCBoYXZlIHRo ZWlyIG5yX3JlcXVlc3RzCj4gY2hhbmdlZCwgc2VlOiBibG9jay9ibGstc3lzZnMuYzpxdWV1ZV9y ZXF1ZXN0c19zdG9yZSgpCj4gCj4gU28gYW55IGJpby1iYXNlZCBETSBkZXZpY2UgKGUuZy4gbGlu ZWFyIHRhcmdldCkgd29uJ3QgYmUgYWxsb3dlZCB0bwo+IGNoYW5nZSBucl9yZXF1ZXN0cy4KPiAK PiBUaGUgb25seSB3YXkgdG8gY2hhbmdlIHRoaXMgd291bGQgYmUgZnJvbSB0aGUgYm90dG9tIHVw Ogo+IDEpIGNoYW5nZSB0aGUgdW5kZXJseWluZyByZXF1ZXN0LWJhc2VkIGRldmljZShzKSAoZS5n LiAvZGV2L3NkYSkKPiAyKSBkbXNldHVwIHN1c3BlbmQgKyBkbXNldHVwIHJlc3VtZSBhbnkgYmlv LWJhc2VkIERNIGRldmljZShzKSB0aGF0IGFyZQo+ICAgIHN0YWNrZWQgb250b3Agb2YgdGhlIHJl cXVlc3QtYmFzZWQgZGV2aWNlKHMpIHlvdSBjaGFuZ2VkIGluIDEpCj4gICAgLS0gdGhpcyB3aWxs IHJlc3RhY2sgdGhlIHF1ZXVlX2xpbWl0cyBmcm9tIHRoZSBib3R0b20gdXA7IHNvIHRoZSBETQo+ ICAgIGRldmljZSB3aWxsIHRoZW4gcmVmbGVjdCB0aGUgdW5kZXJseWluZyBkZXZpY2VzJyBsaW1p dHMuCgpJIHdhcyBtaXN0YWtlbi4gIERNIGNvcmUgKGluIHRoZSBrZXJuZWwpIHdpbGwgb25seSBy ZXN0YWNrIHRoZQpxdWV1ZV9saW1pdHMgKG5yX3JlcXVlc3RzIGluY2x1ZGVkKSBpZiBhIG5ldyBE TSB0YWJsZSBpcyBsb2FkZWQuCgpTbyBhIHNpbXBsZSBzdXNwZW5kK3Jlc3VtZSB3aWxsIF9ub3Rf IGNoYW5nZSB0aGUgYmlvLWJhc2VkIERNIGRldmljZSdzCnF1ZXVlX2xpbWl0czoKCltyb290QHRo ZWdvYXQgfl0jIGxzYmxrIC9kZXYvbnVsbGIwCk5BTUUgICBNQUo6TUlOIFJNICBTSVpFIFJPIFRZ UEUgTU9VTlRQT0lOVApudWxsYjAgMjUzOjAgICAgMCAgMjUwRyAgMCBkaXNrCuKUlOKUgGZvbyAg MjUyOjAgICAgMCAgMjUwRyAgMCBkbQpbcm9vdEB0aGVnb2F0IH5dIyBjYXQgL3N5cy9ibG9jay9u dWxsYjAvcXVldWUvbnJfcmVxdWVzdHMKMTI4Cltyb290QHRoZWdvYXQgfl0jIGNhdCAvc3lzL2Js b2NrL2RtLTAvcXVldWUvbnJfcmVxdWVzdHMKMTI4Cltyb290QHRoZWdvYXQgfl0jIGVjaG8gNjQg PiAvc3lzL2Jsb2NrL251bGxiMC9xdWV1ZS9ucl9yZXF1ZXN0cwpbcm9vdEB0aGVnb2F0IH5dIyBj YXQgL3N5cy9ibG9jay9udWxsYjAvcXVldWUvbnJfcmVxdWVzdHMKNjQKW3Jvb3RAdGhlZ29hdCB+ XSMgY2F0IC9zeXMvYmxvY2svZG0tMC9xdWV1ZS9ucl9yZXF1ZXN0cwoxMjgKW3Jvb3RAdGhlZ29h dCB+XSMgZG1zZXR1cCBzdXNwZW5kIGZvbwpbcm9vdEB0aGVnb2F0IH5dIyBkbXNldHVwIHJlc3Vt ZSBmb28KW3Jvb3RAdGhlZ29hdCB+XSMgY2F0IC9zeXMvYmxvY2svZG0tMC9xdWV1ZS9ucl9yZXF1 ZXN0cwoxMjgKClRoZXJlIHdhcyBzb21lIHJlbGF0ZWQgd29yayBzdGFydGVkIGluIHRoaXMgYXJl YSBieSBIYW5uZXMgKG5vdyBjYydkKQpkdXJpbmcgTFNGIGxhc3QgbW9udGguICBIZSBhZGRlZCBh IGNoYWluLW5vdGlmaWVyIHRvIERNIGRldmljZXMgc28gdGhhdAppZiBhbnkgdW5kZXJseWluZyBk ZXZpY2VzIChkZXZpY2VzIHJlZmVyZW5jZWQgaW4gYSBETSBkZXZpY2UncyB0YWJsZSkKd2VyZSBj aGFuZ2VkIHRoZW4gdGhlIGVudHVyZSBkZXZpY2Ugc3RhY2sgd291bGQgaGF2ZSBpdHMgbGltaXRz CnJlbG9hZGVkLgoKTm90IHN1cmUgd2hlcmUgSGFubmVzIGVuZGVkIHVwIHdpdGggdGhhdC4gIFdv dWxkIGxvdmUgdG8gZ2V0IGl0IHBvbGlzaGVkCm5vdyBhbmQgc3RhZ2VkIGZvciA0LjE4IGluY2x1 c2lvbi4KCkhhbm5lcz8KCi0tCmRtLWRldmVsIG1haWxpbmcgbGlzdApkbS1kZXZlbEByZWRoYXQu Y29tCmh0dHBzOi8vd3d3LnJlZGhhdC5jb20vbWFpbG1hbi9saXN0aW5mby9kbS1kZXZlbA== From mboxrd@z Thu Jan 1 00:00:00 1970 Date: Mon, 14 May 2018 13:53:54 -0400 From: Mike Snitzer Message-ID: <20180514175354.GA15053@redhat.com> References: <20180514161312.GA14685@redhat.com> MIME-Version: 1.0 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180514161312.GA14685@redhat.com> Subject: Re: [linux-lvm] how to set higher then 128 nr_requests on LV's Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: Content-Type: text/plain; charset="utf-8" To: Pankaj Agarwal Cc: Hannes Reinecke , dm-devel@redhat.com, linux-lvm@redhat.com On Mon, May 14 2018 at 12:13pm -0400, Mike Snitzer wrote: > On Sun, May 13 2018 at 8:47pm -0400, > Pankaj Agarwal wrote: > > > Hi, > > How do i set the nr_request value for LV's as it's not writable like other > > drives on a linux system. > > LV's are set as dm-0 and dm-1 on my system. > > #cat /sys/block/dm-0/queue/nr_requests > > 128 > > # echo 256 > /sys/block/dm-0/queue/nr_requests > > -bash: echo: write error: Invalid argument > > In the block layer, bio-based devices cannot have their nr_requests > changed, see: block/blk-sysfs.c:queue_requests_store() > > So any bio-based DM device (e.g. linear target) won't be allowed to > change nr_requests. > > The only way to change this would be from the bottom up: > 1) change the underlying request-based device(s) (e.g. /dev/sda) > 2) dmsetup suspend + dmsetup resume any bio-based DM device(s) that are > stacked ontop of the request-based device(s) you changed in 1) > -- this will restack the queue_limits from the bottom up; so the DM > device will then reflect the underlying devices' limits. I was mistaken. DM core (in the kernel) will only restack the queue_limits (nr_requests included) if a new DM table is loaded. So a simple suspend+resume will _not_ change the bio-based DM device's queue_limits: [root@thegoat ~]# lsblk /dev/nullb0 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nullb0 253:0 0 250G 0 disk └─foo 252:0 0 250G 0 dm [root@thegoat ~]# cat /sys/block/nullb0/queue/nr_requests 128 [root@thegoat ~]# cat /sys/block/dm-0/queue/nr_requests 128 [root@thegoat ~]# echo 64 > /sys/block/nullb0/queue/nr_requests [root@thegoat ~]# cat /sys/block/nullb0/queue/nr_requests 64 [root@thegoat ~]# cat /sys/block/dm-0/queue/nr_requests 128 [root@thegoat ~]# dmsetup suspend foo [root@thegoat ~]# dmsetup resume foo [root@thegoat ~]# cat /sys/block/dm-0/queue/nr_requests 128 There was some related work started in this area by Hannes (now cc'd) during LSF last month. He added a chain-notifier to DM devices so that if any underlying devices (devices referenced in a DM device's table) were changed then the enture device stack would have its limits reloaded. Not sure where Hannes ended up with that. Would love to get it polished now and staged for 4.18 inclusion. Hannes?