v4.16-rc1 + dm-mpath + BFQ

v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

SGVsbG8gUGFvbG8sDQoNCklmIEkgZW5hYmxlIHRoZSBCRlEgc2NoZWR1bGVyIGZvciBhIGRtLW1w
YXRoIGRldmljZSB0aGVuIGEga2VybmVsIG9vcHMNCmFwcGVhcnMgKHNlZSBhbHNvIGJlbG93KS4g
VGhpcyBoYXBwZW5zIHN5c3RlbWF0aWNhbGx5IHdpdGggTGludXMnIHRyZWUgZnJvbQ0KdGhpcyBt
b3JuaW5nIChjb21taXQgNTRjZTY4NWNhZTMwKSBtZXJnZWQgd2l0aCBKZW5zJyBmb3ItbGludXMg
YnJhbmNoIChjb21taXQNCmE3ODc3MzkwNjE0NyAoImJsb2NrLCBiZnE6IGFkZCByZXF1ZXVlLXJl
cXVlc3QgaG9vayIpKSBhbmQgZm9yLW5leHQgYnJhbmNoDQooY29tbWl0IDg4NDU1YWQ3ZjkyOCku
IElzIHRoaXMgYSBrbm93biBpc3N1ZT8NCg0KVGhhbmtzLA0KDQpCYXJ0Lg0KDQpCVUc6IHVuYWJs
ZSB0byBoYW5kbGUga2VybmVsIE5VTEwgcG9pbnRlciBkZXJlZmVyZW5jZSBhdCAwMDAwMDAwMDAw
MDAwMjAwDQpJUDogcmJfZXJhc2UrMHgyODQvMHgzODANClBHRCAwIFA0RCAwIA0KT29wczogMDAw
MiBbIzFdIFBSRUVNUFQgU01QDQpDUFU6IDEgUElEOiAwIENvbW06IHN3YXBwZXIvMSBUYWludGVk
OiBHICAgICAgICBXICAgICAgICA0LjE1LjAtZGJnKyAjMQ0KSGFyZHdhcmUgbmFtZTogUUVNVSBT
dGFuZGFyZCBQQyAoaTQ0MEZYICsgUElJWCwgMTk5NiksIEJJT1MgMS4wLjAtcHJlYnVpbHQucWVt
dS1wcm9qZWN0Lm9yZyAwNC8wMS8yMDE0DQpSSVA6IDAwMTA6cmJfZXJhc2UrMHgyODQvMHgzODAN
CkNhbGwgVHJhY2U6DQogPElSUT4NCiBlbHZfcmJfZGVsKzB4MjAvMHgzMA0KIGJmcV9yZW1vdmVf
cmVxdWVzdCsweDhkLzB4MmUwIFtiZnFdDQogYmZxX2ZpbmlzaF9yZXF1ZXVlX3JlcXVlc3QrMHgy
YmIvMHgzOTAgW2JmcV0NCiBibGtfbXFfZnJlZV9yZXF1ZXN0KzB4NTEvMHgxNzANCiBkbV9zb2Z0
aXJxX2RvbmUrMHhkNS8weDI0MCBbZG1fbW9kXQ0KIGZsdXNoX3NtcF9jYWxsX2Z1bmN0aW9uX3F1
ZXVlKzB4OTIvMHgxNDANCiBzbXBfY2FsbF9mdW5jdGlvbl9zaW5nbGVfaW50ZXJydXB0KzB4NDcv
MHgyYjANCiBjYWxsX2Z1bmN0aW9uX3NpbmdsZV9pbnRlcnJ1cHQrMHhhMi8weGIwDQogPC9JUlE+

Re: v4.16-rc1 + dm-mpath + BFQ

[Jens Axboe]

On 2/9/18 11:54 AM, Bart Van Assche wrote:
> Hello Paolo,
> 
> If I enable the BFQ scheduler for a dm-mpath device then a kernel oops
> appears (see also below). This happens systematically with Linus' tree from
> this morning (commit 54ce685cae30) merged with Jens' for-linus branch (commit
> a78773906147 ("block, bfq: add requeue-request hook")) and for-next branch
> (commit 88455ad7f928). Is this a known issue?

Does it happen on Linus -git as well, or just with my for-linus merged in?
What I'm getting at is if a78773906147 caused this or not.



-- 
Jens Axboe

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

On 02/09/18 10:58, Jens Axboe wrote:
> On 2/9/18 11:54 AM, Bart Van Assche wrote:
>> Hello Paolo,
>>
>> If I enable the BFQ scheduler for a dm-mpath device then a kernel oops
>> appears (see also below). This happens systematically with Linus' tree from
>> this morning (commit 54ce685cae30) merged with Jens' for-linus branch (commit
>> a78773906147 ("block, bfq: add requeue-request hook")) and for-next branch
>> (commit 88455ad7f928). Is this a known issue?
> 
> Does it happen on Linus -git as well, or just with my for-linus merged in?
> What I'm getting at is if a78773906147 caused this or not.

Hello Jens,

Thanks for chiming in. After having reverted commit a78773906147, after 
having rebuilt the BFQ scheduler, after having rebooted and after having 
repeated the test I see the same kernel oops being reported. I think 
that means that this regression is not caused by commit a78773906147. In 
case it would be useful, here is how gdb translates the crash address:

$ gdb block/bfq*ko
(gdb) list *(bfq_remove_request+0x8d)
0x280d is in bfq_remove_request (block/bfq-iosched.c:1760).
1755                    list_del_init(&rq->queuelist);
1756            bfqq->queued[sync]--;
1757            bfqd->queued--;
1758            elv_rb_del(&bfqq->sort_list, rq);
1759
1760            elv_rqhash_del(q, rq);
1761            if (q->last_merge == rq)
1762                    q->last_merge = NULL;
1763
1764            if (RB_EMPTY_ROOT(&bfqq->sort_list)) {

Bart.

Re: v4.16-rc1 + dm-mpath + BFQ

[Jens Axboe]

On 2/9/18 12:14 PM, Bart Van Assche wrote:
> On 02/09/18 10:58, Jens Axboe wrote:
>> On 2/9/18 11:54 AM, Bart Van Assche wrote:
>>> Hello Paolo,
>>>
>>> If I enable the BFQ scheduler for a dm-mpath device then a kernel oops
>>> appears (see also below). This happens systematically with Linus' tree from
>>> this morning (commit 54ce685cae30) merged with Jens' for-linus branch (commit
>>> a78773906147 ("block, bfq: add requeue-request hook")) and for-next branch
>>> (commit 88455ad7f928). Is this a known issue?
>>
>> Does it happen on Linus -git as well, or just with my for-linus merged in?
>> What I'm getting at is if a78773906147 caused this or not.
> 
> Hello Jens,
> 
> Thanks for chiming in. After having reverted commit a78773906147, after 
> having rebuilt the BFQ scheduler, after having rebooted and after having 
> repeated the test I see the same kernel oops being reported. I think 
> that means that this regression is not caused by commit a78773906147. In 
> case it would be useful, here is how gdb translates the crash address:
> 
> $ gdb block/bfq*ko
> (gdb) list *(bfq_remove_request+0x8d)
> 0x280d is in bfq_remove_request (block/bfq-iosched.c:1760).
> 1755                    list_del_init(&rq->queuelist);
> 1756            bfqq->queued[sync]--;
> 1757            bfqd->queued--;
> 1758            elv_rb_del(&bfqq->sort_list, rq);
> 1759
> 1760            elv_rqhash_del(q, rq);
> 1761            if (q->last_merge == rq)
> 1762                    q->last_merge = NULL;
> 1763
> 1764            if (RB_EMPTY_ROOT(&bfqq->sort_list)) {

Looks very odd. So clearly RQF_HASHED is set, but we're blowing up on
the hash list pointers. I'll let Paolo take a look at this one. Thanks
for testing without that commit, I want to push out my pending fixes
today and this would have thrown a wrench in the works.

-- 
Jens Axboe

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 09 feb 2018, alle ore 20:18, Jens Axboe <axboe@kernel.dk> ha =
scritto:
>=20
> On 2/9/18 12:14 PM, Bart Van Assche wrote:
>> On 02/09/18 10:58, Jens Axboe wrote:
>>> On 2/9/18 11:54 AM, Bart Van Assche wrote:
>>>> Hello Paolo,
>>>>=20
>>>> If I enable the BFQ scheduler for a dm-mpath device then a kernel =
oops
>>>> appears (see also below). This happens systematically with Linus' =
tree from
>>>> this morning (commit 54ce685cae30) merged with Jens' for-linus =
branch (commit
>>>> a78773906147 ("block, bfq: add requeue-request hook")) and for-next =
branch
>>>> (commit 88455ad7f928). Is this a known issue?
>>>=20
>>> Does it happen on Linus -git as well, or just with my for-linus =
merged in?
>>> What I'm getting at is if a78773906147 caused this or not.
>>=20
>> Hello Jens,
>>=20
>> Thanks for chiming in. After having reverted commit a78773906147, =
after=20
>> having rebuilt the BFQ scheduler, after having rebooted and after =
having=20
>> repeated the test I see the same kernel oops being reported. I think=20=

>> that means that this regression is not caused by commit a78773906147. =
In=20
>> case it would be useful, here is how gdb translates the crash =
address:
>>=20
>> $ gdb block/bfq*ko
>> (gdb) list *(bfq_remove_request+0x8d)
>> 0x280d is in bfq_remove_request (block/bfq-iosched.c:1760).
>> 1755                    list_del_init(&rq->queuelist);
>> 1756            bfqq->queued[sync]--;
>> 1757            bfqd->queued--;
>> 1758            elv_rb_del(&bfqq->sort_list, rq);
>> 1759
>> 1760            elv_rqhash_del(q, rq);
>> 1761            if (q->last_merge =3D=3D rq)
>> 1762                    q->last_merge =3D NULL;
>> 1763
>> 1764            if (RB_EMPTY_ROOT(&bfqq->sort_list)) {
>=20
> Looks very odd. So clearly RQF_HASHED is set, but we're blowing up on
> the hash list pointers. I'll let Paolo take a look at this one. Thanks
> for testing without that commit, I want to push out my pending fixes
> today and this would have thrown a wrench in the works.
>=20

Also this smells a little bit like some spurious elevator call.
Unfortunately I have no clue on the cause.  To go on, I need at least
to reproduce it.  In this respect: Bart, could you please tell me how
to setup the offending configuration, and to cause the failure?
Possibly with just one, or at most two PCs.  I don't have fancier hw
at the moment.

Thanks,
Paolo

> --=20
> Jens Axboe

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

On 02/11/18 23:35, Paolo Valente wrote:
> Also this smells a little bit like some spurious elevator call.
> Unfortunately I have no clue on the cause.  To go on, I need at least
> to reproduce it.  In this respect: Bart, could you please tell me how
> to setup the offending configuration, and to cause the failure?
> Possibly with just one, or at most two PCs.  I don't have fancier hw
> at the moment.

Hello Paolo,

Although I expect that it is possible to reproduce this with an 
unmodified v4.16-rc1 kernel, this is how I ran into this issue:
* Clone the for-next branch of https://github.com/bvanassche/linux.
* Build and install that kernel in a virtual machine.
* Clone https://github.com/bvanassche/srp-test.
* Run the following command:
   srp-test/run_tests -c -d -r 10 -t 02-mq -e bfq

Thanks,

Bart.

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 12 feb 2018, alle ore 17:31, Bart Van Assche =
<bart.vanassche@wdc.com> ha scritto:
>=20
> On 02/11/18 23:35, Paolo Valente wrote:
>> Also this smells a little bit like some spurious elevator call.
>> Unfortunately I have no clue on the cause.  To go on, I need at least
>> to reproduce it.  In this respect: Bart, could you please tell me how
>> to setup the offending configuration, and to cause the failure?
>> Possibly with just one, or at most two PCs.  I don't have fancier hw
>> at the moment.
>=20
> Hello Paolo,
>=20
> Although I expect that it is possible to reproduce this with an =
unmodified v4.16-rc1 kernel, this is how I ran into this issue:
> * Clone the for-next branch of https://github.com/bvanassche/linux.
> * Build and install that kernel in a virtual machine.
> * Clone https://github.com/bvanassche/srp-test.
> * Run the following command:
>  srp-test/run_tests -c -d -r 10 -t 02-mq -e bfq
>=20

Hi Bart,
as a first attempt, I've followed your steps, but got:
Error: could not find sg_reset
expectedly because of dependencies that you are implying in your steps.

So, I have followed the instructions in the srp-test README for the
case "Running the Tests on an Ethernet Setup", directly on a 4.16-rc1.

For ib_srp-backport, I get a lot of warnings like the following one,
at "make install" (preceded by corresponding warnings at the end of
the compilation):
depmod: WARNING: /lib/modules/4.16.0-rc1+/extra/ib_srp.ko needs unknown =
symbol rdma_resolve_addr

Unfortunately, it gets worse while executing "make scst srpt":

  CC [M]  /home/paolo/scst/srpt/src/ib_srpt.o
In file included from /home/paolo/scst/srpt/src/ib_srpt.c:62:0:
/home/paolo/scst/srpt/src/ib_srpt.h:481:8: error: redefinition of =
=E2=80=98struct srp_login_req_rdma=E2=80=99
 struct srp_login_req_rdma {
        ^~~~~~~~~~~~~~~~~~
In file included from /home/paolo/scst/srpt/src/ib_srpt.h:44:0,
                 from /home/paolo/scst/srpt/src/ib_srpt.c:62:
/mnt/linux-dev/linux/include/scsi/srp.h:139:8: note: originally defined =
here
 struct srp_login_req_rdma {
        ^~~~~~~~~~~~~~~~~~

Could you please give me some help, so as to not get lost among these =
issues?

Thanks,
Paolo

> Thanks,
>=20
> Bart.

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

T24gVHVlLCAyMDE4LTAyLTEzIGF0IDE5OjM4ICswMTAwLCBQYW9sbyBWYWxlbnRlIHdyb3RlOg0K
PiBhcyBhIGZpcnN0IGF0dGVtcHQsIEkndmUgZm9sbG93ZWQgeW91ciBzdGVwcywgYnV0IGdvdDoN
Cj4gRXJyb3I6IGNvdWxkIG5vdCBmaW5kIHNnX3Jlc2V0DQoNClBsZWFzZSBpbnN0YWxsIHRoZSBz
ZzNfdXRpbHMgcGFja2FnZS4gRXZlcnkgTGludXggZGlzdHJvIEkga25vdyBvZiBzdXBwb3J0cyB0
aGF0DQpwYWNrYWdlLiBBbmQgaW4gY2FzZSB5b3Ugd291bGQgbGlrZSB0byBpbnN0YWxsIGl0IGZy
b20gc291cmNlLCB0aGUgc291cmNlIGNvZGUgb2YNCnRoYXQgcGFja2FnZSBpcyBhdmFpbGFibGUg
ZnJvbSBodHRwOi8vc2cuZGFubnkuY3ovc2cvc2czX3V0aWxzLmh0bWwuDQoNCj4gRm9yIGliX3Ny
cC1iYWNrcG9ydCwgSSBnZXQgYSBsb3Qgb2Ygd2FybmluZ3MgbGlrZSB0aGUgZm9sbG93aW5nIG9u
ZSwNCj4gYXQgIm1ha2UgaW5zdGFsbCIgKHByZWNlZGVkIGJ5IGNvcnJlc3BvbmRpbmcgd2Fybmlu
Z3MgYXQgdGhlIGVuZCBvZg0KPiB0aGUgY29tcGlsYXRpb24pOg0KPiBkZXBtb2Q6IFdBUk5JTkc6
IC9saWIvbW9kdWxlcy80LjE2LjAtcmMxKy9leHRyYS9pYl9zcnAua28gbmVlZHMgdW5rbm93biBz
eW1ib2wgcmRtYV9yZXNvbHZlX2FkZHINCj4gDQo+IFVuZm9ydHVuYXRlbHksIGl0IGdldHMgd29y
c2Ugd2hpbGUgZXhlY3V0aW5nICJtYWtlIHNjc3Qgc3JwdCI6DQoNClBsZWFzZSBuZWl0aGVyIGlu
c3RhbGwgdGhlIGliX3NycC1iYWNrcG9ydCBkcml2ZXIgbm9yIFNDU1QuIFRoZXNlIGRyaXZlcnMg
aGF2ZQ0Kbm90IHlldCBiZWVuIHRlc3RlZCBhZ2FpbnN0IGtlcm5lbCB2NC4xNi1yYzEuIEkgcHJv
dmlkZWQgeW91IGEga2VybmVsIHRyZWUgaW4NCndoaWNoIGJvdGggdGhlIFNSUCBpbml0aWF0b3Ig
YW5kIHRhcmdldCBkcml2ZXJzIHN1cHBvcnQgUm9DRSBzdWNoIHRoYXQgeW91IGRvbid0DQpuZWVk
IHRvIGluc3RhbGwgdGhlc2Ugb3V0LW9mLXRyZWUgZHJpdmVycy4gSSB0aGluayBhbGwgdGhhdCB5
b3UgbmVlZCBmcm9tIHRoZQ0Kc3JwLXRlc3QgUkVBRE1FIGRvY3VtZW50IGFyZSB0aGUgaW5zdHJ1
Y3Rpb25zIHRvIGNvbmZpZ3VyZSAvZXRjL211bHRpcGF0aC5jb25mDQphbmQgdGhlIGluc3RydWN0
aW9ucyBmb3IgaW5zdGFsbGluZyB0aGUgcmVxdWlyZWQgcGFja2FnZXMuIEZyb20gdGhhdCBSRUFE
TUUNCmRvY3VtZW50Og0KDQpJbnN0YWxsIHRoZSBmb2xsb3dpbmcgc29mdHdhcmUgcGFja2FnZXMg
aWYgdGhlc2UgaGF2ZSBub3QgeWV0IGJlZW4gaW5zdGFsbGVkOg0KZmlvLCBnY2MtYysrLCBtYWtl
LCBtdWx0aXBhdGgtdG9vbHMgb3IgZGV2aWNlLW1hcHBlci1tdWx0aXBhdGgsIHNnM191dGlscywN
CnNycHRvb2xzLCBlMmZzcHJvZ3MgYW5kIHhmc3Byb2dzLg0KDQpUaGFua3MsDQoNCkJhcnQuDQoN
Cg0K

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 13 feb 2018, alle ore 19:47, Bart Van Assche =
<bart.vanassche@wdc.com> ha scritto:
>=20
> On Tue, 2018-02-13 at 19:38 +0100, Paolo Valente wrote:
>> as a first attempt, I've followed your steps, but got:
>> Error: could not find sg_reset
>=20
> Please install the sg3_utils package. Every Linux distro I know of =
supports that
> package.

I happened to do this test on a Fedora.

> And in case you would like to install it from source, the source code =
of
> that package is available from http://sg.danny.cz/sg/sg3_utils.html.
>=20
>> For ib_srp-backport, I get a lot of warnings like the following one,
>> at "make install" (preceded by corresponding warnings at the end of
>> the compilation):
>> depmod: WARNING: /lib/modules/4.16.0-rc1+/extra/ib_srp.ko needs =
unknown symbol rdma_resolve_addr
>>=20
>> Unfortunately, it gets worse while executing "make scst srpt":
>=20
> Please neither install the ib_srp-backport driver nor SCST. These =
drivers have
> not yet been tested against kernel v4.16-rc1. I provided you a kernel =
tree in
> which both the SRP initiator and target drivers support RoCE such that =
you don't
> need to install these out-of-tree drivers. I think all that you need =
from the
> srp-test README document are the instructions to configure =
/etc/multipath.conf
> and the instructions for installing the required packages. =46rom that =
README
> document:
>=20
> Install the following software packages if these have not yet been =
installed:
> fio, gcc-c++, make, multipath-tools or device-mapper-multipath, =
sg3_utils,
> srptools, e2fsprogs and xfsprogs.
>=20

Thank you very much for these instructions Bart.

After following all of them (and taking some other step needed), I
invoked:
sudo ./run_tests -c -d -r 10 -t 02-mq -e bfq

But I got the following:
./lib/functions: riga 34: /sys/class/block/ram0/size: No such file or =
directory
./lib/functions: riga 34: * 512: errore di sintassi: atteso un operando =
(il token dell'errore =C3=A8 "* 512")
Unloaded the ib_srpt kernel module
Unloaded the rdma_rxe kernel module
modprobe: FATAL: Module ib_uverbs not found in directory =
/lib/modules/4.16.0-rc1+
modprobe: FATAL: Module ib_umad not found in directory =
/lib/modules/4.16.0-rc1+
SoftRoCE network interfaces: rxe0
modprobe: FATAL: Module target_core_iblock not found in directory =
/lib/modules/4.16.0-rc1+

So I think I need a little more help, to have this working in a
reasonable amount of time.  In particular, could you tell me all what
is missing?

Thanks,
Paolo

> Thanks,
>=20
> Bart.
>=20
>=20

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

On 02/14/18 09:55, Paolo Valente wrote:
> After following all of them (and taking some other step needed), I
> invoked:
> sudo ./run_tests -c -d -r 10 -t 02-mq -e bfq
> 
> But I got the following:
> ./lib/functions: riga 34: /sys/class/block/ram0/size: No such file or directory
> ./lib/functions: riga 34: * 512: errore di sintassi: atteso un operando (il token dell'errore è "* 512")
> Unloaded the ib_srpt kernel module
> Unloaded the rdma_rxe kernel module
> modprobe: FATAL: Module ib_uverbs not found in directory /lib/modules/4.16.0-rc1+
> modprobe: FATAL: Module ib_umad not found in directory /lib/modules/4.16.0-rc1+
> SoftRoCE network interfaces: rxe0
> modprobe: FATAL: Module target_core_iblock not found in directory /lib/modules/4.16.0-rc1+
> 
> So I think I need a little more help, to have this working in a
> reasonable amount of time.  In particular, could you tell me all what
> is missing?

Hello Paolo,

Can you check whether CONFIG_BLK_DEV_RAM, CONFIG_INFINIBAND, 
CONFIG_INFINIBAND_USER_MAD, CONFIG_INFINIBAND_USER_ACCESS, 
CONFIG_INFINIBAND_USER_MEM, CONFIG_INFINIBAND_IPOIB, 
CONFIG_INFINIBAND_SRP, CONFIG_INFINIBAND_SRPT and CONFIG_RDMA_RXE were 
enabled in your kernel config?

Thanks,

Bart.

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 14 feb 2018, alle ore 19:11, Bart Van Assche =
<bart.vanassche@wdc.com> ha scritto:
>=20
> On 02/14/18 09:55, Paolo Valente wrote:
>> After following all of them (and taking some other step needed), I
>> invoked:
>> sudo ./run_tests -c -d -r 10 -t 02-mq -e bfq
>> But I got the following:
>> ./lib/functions: riga 34: /sys/class/block/ram0/size: No such file or =
directory
>> ./lib/functions: riga 34: * 512: errore di sintassi: atteso un =
operando (il token dell'errore =C3=A8 "* 512")
>> Unloaded the ib_srpt kernel module
>> Unloaded the rdma_rxe kernel module
>> modprobe: FATAL: Module ib_uverbs not found in directory =
/lib/modules/4.16.0-rc1+
>> modprobe: FATAL: Module ib_umad not found in directory =
/lib/modules/4.16.0-rc1+
>> SoftRoCE network interfaces: rxe0
>> modprobe: FATAL: Module target_core_iblock not found in directory =
/lib/modules/4.16.0-rc1+
>> So I think I need a little more help, to have this working in a
>> reasonable amount of time.  In particular, could you tell me all what
>> is missing?
>=20
> Hello Paolo,
>=20
> Can you check whether CONFIG_BLK_DEV_RAM, CONFIG_INFINIBAND, =
CONFIG_INFINIBAND_USER_MAD, CONFIG_INFINIBAND_USER_ACCESS, =
CONFIG_INFINIBAND_USER_MEM, CONFIG_INFINIBAND_IPOIB, =
CONFIG_INFINIBAND_SRP, CONFIG_INFINIBAND_SRPT and CONFIG_RDMA_RXE were =
enabled in your kernel config?
>=20

(+Linus, Ulf)

Hi Bart,
after enabling the listing options in your list, and a few other
related options, such iblock support, I get this:

$ sudo ./run_tests -c -d -r 10 -t 02-mq -e bfq
Unloaded the ib_srpt kernel module
Unloaded the rdma_rxe kernel module
SoftRoCE network interfaces: rxe0
Zero-initializing /dev/ram0 ... done
Zero-initializing /dev/ram1 ... done
mkdir: impossibile creare la directory "021c:42ff:fe4c:fac9": Invalid =
argument
Retrying with old port name format
mkdir: impossibile creare la directory =
"0xfe80000000000000021c42fffe4cfac9": Invalid argument

Thanks for your patience and collaboration,
Paolo

> Thanks,
>=20
> Bart.

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

T24gRnJpLCAyMDE4LTAyLTE2IGF0IDA4OjM5ICswMTAwLCBQYW9sbyBWYWxlbnRlIHdyb3RlOg0K
PiBhZnRlciBlbmFibGluZyB0aGUgbGlzdGluZyBvcHRpb25zIGluIHlvdXIgbGlzdCwgYW5kIGEg
ZmV3IG90aGVyDQo+IHJlbGF0ZWQgb3B0aW9ucywgc3VjaCBpYmxvY2sgc3VwcG9ydCwgSSBnZXQg
dGhpczoNCj4gDQo+ICQgc3VkbyAuL3J1bl90ZXN0cyAtYyAtZCAtciAxMCAtdCAwMi1tcSAtZSBi
ZnENCj4gVW5sb2FkZWQgdGhlIGliX3NycHQga2VybmVsIG1vZHVsZQ0KPiBVbmxvYWRlZCB0aGUg
cmRtYV9yeGUga2VybmVsIG1vZHVsZQ0KPiBTb2Z0Um9DRSBuZXR3b3JrIGludGVyZmFjZXM6IHJ4
ZTANCj4gWmVyby1pbml0aWFsaXppbmcgL2Rldi9yYW0wIC4uLiBkb25lDQo+IFplcm8taW5pdGlh
bGl6aW5nIC9kZXYvcmFtMSAuLi4gZG9uZQ0KPiBta2RpcjogaW1wb3NzaWJpbGUgY3JlYXJlIGxh
IGRpcmVjdG9yeSAiMDIxYzo0MmZmOmZlNGM6ZmFjOSI6IEludmFsaWQgYXJndW1lbnQNCj4gUmV0
cnlpbmcgd2l0aCBvbGQgcG9ydCBuYW1lIGZvcm1hdA0KPiBta2RpcjogaW1wb3NzaWJpbGUgY3Jl
YXJlIGxhIGRpcmVjdG9yeSAiMHhmZTgwMDAwMDAwMDAwMDAwMDIxYzQyZmZmZTRjZmFjOSI6IElu
dmFsaWQgYXJndW1lbnQNCg0KSGVsbG8gUGFvbG8sDQoNClRoYXQgcHJvYmFibHkgbWVhbnMgdGhh
dCB0aGVyZSBpcyBzdGlsbCBzb21ldGhpbmcgbWlzc2luZyBmcm9tIHRoZSBrZXJuZWwNCmNvbmZp
ZyB0aGF0IHlvdSBhcmUgdXNpbmcuIFBsZWFzZSBzZW5kIHRoYXQga2VybmVsLWNvbmZpZyB0byBt
ZSAob2ZmLWxpc3QpDQpzdWNoIHRoYXQgSSBjYW4gaGF2ZSBhIGxvb2sgYXQgaXQuDQoNClRoYW5r
cywNCg0KQmFydC4NCg0KDQoNCg==

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

T24gRnJpLCAyMDE4LTAyLTE2IGF0IDA4OjM5ICswMTAwLCBQYW9sbyBWYWxlbnRlIHdyb3RlOg0K
PiBhZnRlciBlbmFibGluZyB0aGUgbGlzdGluZyBvcHRpb25zIGluIHlvdXIgbGlzdCwgYW5kIGEg
ZmV3IG90aGVyDQo+IHJlbGF0ZWQgb3B0aW9ucywgc3VjaCBpYmxvY2sgc3VwcG9ydCwgSSBnZXQg
dGhpczoNCj4gDQo+ICQgc3VkbyAuL3J1bl90ZXN0cyAtYyAtZCAtciAxMCAtdCAwMi1tcSAtZSBi
ZnENCj4gVW5sb2FkZWQgdGhlIGliX3NycHQga2VybmVsIG1vZHVsZQ0KPiBVbmxvYWRlZCB0aGUg
cmRtYV9yeGUga2VybmVsIG1vZHVsZQ0KPiBTb2Z0Um9DRSBuZXR3b3JrIGludGVyZmFjZXM6IHJ4
ZTANCj4gWmVyby1pbml0aWFsaXppbmcgL2Rldi9yYW0wIC4uLiBkb25lDQo+IFplcm8taW5pdGlh
bGl6aW5nIC9kZXYvcmFtMSAuLi4gZG9uZQ0KPiBta2RpcjogaW1wb3NzaWJpbGUgY3JlYXJlIGxh
IGRpcmVjdG9yeSAiMDIxYzo0MmZmOmZlNGM6ZmFjOSI6IEludmFsaWQgYXJndW1lbnQNCj4gUmV0
cnlpbmcgd2l0aCBvbGQgcG9ydCBuYW1lIGZvcm1hdA0KPiBta2RpcjogaW1wb3NzaWJpbGUgY3Jl
YXJlIGxhIGRpcmVjdG9yeSAiMHhmZTgwMDAwMDAwMDAwMDAwMDIxYzQyZmZmZTRjZmFjOSI6IElu
dmFsaWQgYXJndW1lbnQNCg0KSGVsbG8gUGFvbG8sDQoNClRoYW5rIHlvdSBmb3IgaGF2aW5nIHNo
YXJlZCB5b3VyIGtlcm5lbCBjb25maWcgb2ZmLWxpc3QuIEFmdGVyIGhhdmluZw0KbWFkZSB0aGUg
Zm9sbG93aW5nIGNoYW5nZXMgdG8geW91ciBrZXJuZWwgY29uZmlnIEkgd2FzIGFibGUgdG8gcnVu
IHRoZQ0Kc3JwLXRlc3Qgc29mdHdhcmU6DQoqIEVuYWJsZSBDT05GSUdfRE1fTVVMVElQQVRIX1FM
LCBDT05GSUdfRE1fTVVMVElQQVRIX1NULA0KICBDT05GSUdfU0NTSV9ESF9SREFDLCBDT05GSUdf
U0NTSV9ESF9FTUMgYW5kIENPTkZJR19TQ1NJX0RIX0FMVUEuDQoqIERpc2FibGUgQ09ORklHX0tB
U0FOLiBBcHBhcmVudGx5IHRoZXJlIGlzIGFuIGluY29tcGF0aWJpbGl0eSBiZXR3ZWVuIHRoZQ0K
ICByZG1hX3J4ZSBkcml2ZXIgYW5kIEtBU0FOLiBJJ20gc3RpbGwgYW5hbHl6aW5nIHRoaXMuDQoN
ClBsZWFzZSBsZXQgbWUga25vdyB3aGV0aGVyIHRoZXNlIGNoYW5nZXMgYWxzbyBhbGxvdyB5b3Ug
dG8gcnVuIHRoZSBzcnAtdGVzdA0Kc29mdHdhcmUgYW5kIHdoZXRoZXIgeW91IGNhbiByZXByb2R1
Y2Ugd2hhdCBJIHJlcG9ydGVkIGF0IHRoZSBzdGFydCBvZiB0aGlzDQplLW1haWwgdGhyZWFkLg0K
DQpUaGFua3MsDQoNCkJhcnQuDQoNCg0K

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

T24gRnJpLCAyMDE4LTA1LTA0IGF0IDE2OjQyIC0wNDAwLCBMYXVyZW5jZSBPYmVybWFuIHdyb3Rl
Og0KPiBJIHdhcyBuZXZlciBhYmxlIHRvIHJlcHJvZHVjZSBCYXJ0cyBvcmlnaW5hbCBpc3N1ZSB1
c2luZyBoaXMgdHJlZSBhbmQNCj4gYWN0dWFsIG1seDUvY3g0IGhhcmR3YXJlIGFuZCBpYnNycA0K
PiBJIGVuYWJsZWQgQkZRIHdpdGggbm8gb3RoZXIgc3BlY2lhbCB0dW5pbmcgZm9yIHRoZSBtb2F0
aCBhbmQgc3VicGF0aHMuDQo+IEkgd2FzIHdhaXRpbmcgZm9yIGhpbSB0byBjb21lIGJhY2sgZnJv
bSB2YWNhdGlvbiB0byBjaGVjayB3aXRoIGhpbS4NCg0KKGJhY2sgaW4gdGhlIG9mZmljZSkNCg0K
SGVsbG8gTGF1cmVuY2UsDQoNCldoYXQgSSB1bmRlcnN0b29kIGZyb20gb2ZmLWxpc3QgY29tbXVu
aWNhdGlvbiBpcyB0aGF0IHlvdSB0cmllZCB0byBmaW5kDQphIHdheSB0byByZXByb2R1Y2Ugd2hh
dCBJIHJlcG9ydGVkIHdpdGhvdXQgdXNpbmcgdGhlIHNycC10ZXN0IHNvZnR3YXJlLg0KTXkgdW5k
ZXJzdGFuZGluZyBpcyB0aGF0IGJvdGggUGFvbG8gYW5kIEkgY2FuIHJlcHJvZHVjZSB0aGUgcmVw
b3J0ZWQgaXNzdWUNCndpdGggdGhlIHNycC10ZXN0IHNvZnR3YXJlLg0KDQpCYXJ0Lg0KDQoNCg0K

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 10 mag 2018, alle ore 17:16, Bart Van Assche =
<Bart.VanAssche@wdc.com> ha scritto:
>=20
> On Fri, 2018-05-04 at 16:42 -0400, Laurence Oberman wrote:
>> I was never able to reproduce Barts original issue using his tree and
>> actual mlx5/cx4 hardware and ibsrp
>> I enabled BFQ with no other special tuning for the moath and =
subpaths.
>> I was waiting for him to come back from vacation to check with him.
>=20
> (back in the office)
>=20
> Hello Laurence,
>=20
> What I understood from off-list communication is that you tried to =
find
> a way to reproduce what I reported without using the srp-test =
software.
> My understanding is that both Paolo and I can reproduce the reported =
issue
> with the srp-test software.
>=20

Thanks for chiming in, Bart.

Above all, with my fix [1] it should be gone.

Looking forward to your feedback,
Paolo

[1] =
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1682264.html
> Bart.
>=20
>=20
>=20

Re: v4.16-rc1 + dm-mpath + BFQ

[Laurence Oberman]

On Thu, 2018-05-10 at 15:16 +0000, Bart Van Assche wrote:
> On Fri, 2018-05-04 at 16:42 -0400, Laurence Oberman wrote:
> > I was never able to reproduce Barts original issue using his tree
> > and
> > actual mlx5/cx4 hardware and ibsrp
> > I enabled BFQ with no other special tuning for the moath and
> > subpaths.
> > I was waiting for him to come back from vacation to check with him.
> 
> (back in the office)
> 
> Hello Laurence,
> 
> What I understood from off-list communication is that you tried to
> find
> a way to reproduce what I reported without using the srp-test
> software.
> My understanding is that both Paolo and I can reproduce the reported
> issue
> with the srp-test software.
> 
> Bart.
> 
> 
> 

Hello Bart

using your kernel
4.17.0-rc2.bart+

CONFIG_IOSCHED_BFQ=y
CONFIG_BFQ_GROUP_IOSCHED=y

These are all SRP LUNS

36001405b2b5c6c24c084b6fa4d55da2f dm-27 LIO-ORG ,block-10        
size=3.9G features='2 queue_mode mq' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 2:0:0:9  sdap 66:144 active ready running
  `- 1:0:0:9  sdaz 67:48  active ready running

36001405b26ebe76dcb94a489f6f245f8 dm-18 LIO-ORG ,block-21        
size=3.9G features='2 queue_mode mq' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 2:0:0:20 sdx  65:112 active ready running
  `- 1:0:0:20 sdaa 65:160 active ready running

[root@ibclient ~]# cd /sys/block
[root@ibclient block]# cat /sys/block/dm-18/queue/scheduler
mq-deadline kyber [bfq] none
[root@ibclient block]# cat /sys/block/sdaa/queue/scheduler
mq-deadline kyber [bfq] none
[root@ibclient block]# cat /sys/block/sdx/queue/scheduler
mq-deadline kyber [bfq] none

Not using the test software just exercising the LUNS via my own tests I
am unable to get the OOPS

I guess something in the srp-test software triggers it then.

Doing plenty of IO to 5 mpath devices (1.3Gbytes/sec)

#Time     cpu sys inter  ctxsw Free Buff Cach Inac Slab  Map
KBRead  Reads KBWrit Writes   KBIn  PktIn  KBOut  PktOut 
12:08:32    0   0  1437   1107  88G   5M   1G 902M 300M
178M  1380K    345      0      0      6     74      0       4 

Thanks
Laurence

Re: v4.16-rc1 + dm-mpath + BFQ

[Bart Van Assche]

T24gRnJpLCAyMDE4LTA1LTA0IGF0IDIyOjExICswMjAwLCBQYW9sbyBWYWxlbnRlIHdyb3RlOg0K
PiA+IElsIGdpb3JubyAzMCBtYXIgMjAxOCwgYWxsZSBvcmUgMTg6NTcsIEJhcnQgVmFuIEFzc2No
ZSA8YmFydC52YW5hc3NjaGVAd2RjLmNvbT4gaGEgc2NyaXR0bzoNCj4gPiANCj4gPiBPbiBGcmks
IDIwMTgtMDMtMzAgYXQgMTA6MjMgKzAyMDAsIFBhb2xvIFZhbGVudGUgd3JvdGU6DQo+ID4gPiBT
dGlsbCA0LjE2LXJjMSwgYmVpbmcgdGhhdCB0aGUgdmVyc2lvbiBmb3Igd2hpY2ggeW91IHJlcG9y
dGVkIHRoaXMNCj4gPiA+IGlzc3VlIGluIHRoZSBmaXJzdCBwbGFjZS4NCj4gPiANCj4gPiBBIHZh
bmlsbGEgdjQuMTYtcmMxIGtlcm5lbCBpcyBub3Qgc3VmZmljaWVudCB0byBydW4gdGhlIHNycC10
ZXN0IHNvZnR3YXJlDQo+ID4gc2luY2UgUkRNQS9DTSBzdXBwb3J0IGZvciB0aGUgU1JQIHRhcmdl
dCBkcml2ZXIgaXMgbWlzc2luZyBmcm9tIHRoYXQga2VybmVsLg0KPiA+IFRoYXQncyB3aHkgSSBh
c2tlZCB5b3UgdG8gdXNlIHRoZSBmb3ItbmV4dCBicmFuY2ggZnJvbSBteSBnaXRodWIgcmVwb3Np
dG9yeQ0KPiA+IGluIGEgcHJldmlvdXMgZS1tYWlsLiBBbnl3YXksIHNpbmNlIHRoZSBuZWNlc3Nh
cnkgcGF0Y2hlcyBhcmUgbm93IGluDQo+ID4gbGludXgtbmV4dCwgdGhlIHNycC10ZXN0IHNvZnR3
YXJlIGNhbiBhbHNvIGJlIHJ1biBhZ2FpbnN0IGxpbnV4LW5leHQuIEhlcmUNCj4gPiBhcmUgdGhl
IHJlc3VsdHMgdGhhdCBJIG9idGFpbmVkIHdpdGggbGFiZWwgbmV4dC0yMDE4MDMyOSBhbmQgdGhl
IGtlcm5lbA0KPiA+IGNvbmZpZyBhdHRhY2hlZCB0byB5b3VyIHByZXZpb3VzIGUtbWFpbDoNCj4g
PiANCj4gPiAjIHdoaWxlIC4vc3JwLXRlc3QvcnVuX3Rlc3RzIC1jIC1kIC1yIDEwIC1lIGJmcTsg
ZG8gOjsgZG9uZQ0KPiA+IA0KPiA+IEJVRzogdW5hYmxlIHRvIGhhbmRsZSBrZXJuZWwgTlVMTCBw
b2ludGVyIGRlcmVmZXJlbmNlIGF0IDAwMDAwMDAwMDAwMDAyMDANCj4gPiBQR0QgMCBQNEQgMCAN
Cj4gPiBPb3BzOiAwMDAyIFsjMV0gU01QIFBUSQ0KPiA+IEhhcmR3YXJlIG5hbWU6IFFFTVUgU3Rh
bmRhcmQgUEMgKGk0NDBGWCArIFBJSVgsIDE5OTYpLCBCSU9TIDEuMC4wLXByZWJ1aWx0LnFlbXUt
cHJvamVjdC5vcmcgMDQvMDEvMjAxNA0KPiA+IFJJUDogMDAxMDpyYl9lcmFzZSsweDI4NC8weDM4
MA0KPiA+IENhbGwgVHJhY2U6DQo+ID4gPElSUT4NCj4gPiBlbHZfcmJfZGVsKzB4MjQvMHgzMA0K
PiA+IGJmcV9yZW1vdmVfcmVxdWVzdCsweDlhLzB4MmUwIFtiZnFdDQo+ID4gPyByY3VfcmVhZF9s
b2NrX3NjaGVkX2hlbGQrMHg2NC8weDcwDQo+ID4gPyB1cGRhdGVfbG9hZF9hdmcrMHg3MmIvMHg3
NjANCj4gPiBiZnFfZmluaXNoX3JlcXVldWVfcmVxdWVzdCsweDJlMS8weDNiMCBbYmZxXQ0KPiA+
ID8gX19sb2NrX2lzX2hlbGQrMHg1YS8weGEwDQo+ID4gYmxrX21xX2ZyZWVfcmVxdWVzdCsweDVm
LzB4MWEwDQo+ID4gYmxrX3B1dF9yZXF1ZXN0KzB4MjMvMHg2MA0KPiA+IG11bHRpcGF0aF9yZWxl
YXNlX2Nsb25lKzB4ZS8weDEwDQo+ID4gZG1fc29mdGlycV9kb25lKzB4ZTMvMHgyNzANCj4gPiBf
X2Jsa19tcV9jb21wbGV0ZV9yZXF1ZXN0X3JlbW90ZSsweDE4LzB4MjANCj4gPiBmbHVzaF9zbXBf
Y2FsbF9mdW5jdGlvbl9xdWV1ZSsweGExLzB4MTUwDQo+ID4gZ2VuZXJpY19zbXBfY2FsbF9mdW5j
dGlvbl9zaW5nbGVfaW50ZXJydXB0KzB4MTMvMHgzMA0KPiA+IHNtcF9jYWxsX2Z1bmN0aW9uX3Np
bmdsZV9pbnRlcnJ1cHQrMHg0ZC8weDIyMA0KPiA+IGNhbGxfZnVuY3Rpb25fc2luZ2xlX2ludGVy
cnVwdCsweGYvMHgyMA0KPiA+IDwvSVJRPg0KPiANCj4gSSBzdXNwZWN0IG15IHJlY2VudCBmaXgg
WzFdIG1pZ2h0IGZpeCB5b3VyIGZhaWx1cmUgdG9vLg0KPiANCj4gWzFdIGh0dHBzOi8vd3d3Lm1h
aWwtYXJjaGl2ZS5jb20vbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZy9tc2cxNjgyMjY0Lmh0
bWwNCg0KSGVsbG8gUGFvbG8sDQoNCldpdGggcGF0Y2ggWzFdIGFwcGxpZWQgSSBjYW4ndCByZXBy
b2R1Y2UgdGhlIGFmb3JlbWVudGlvbmVkIGNyYXNoLiBJIHdpbGwgYWRkDQpteSBUZXN0ZWQtYnku
DQoNClRoYW5rcywNCg0KQmFydC4NCg0KDQo=

Re: v4.16-rc1 + dm-mpath + BFQ

[Paolo Valente]

> Il giorno 10 mag 2018, alle ore 18:12, Bart Van Assche =
<Bart.VanAssche@wdc.com> ha scritto:
>=20
> On Fri, 2018-05-04 at 22:11 +0200, Paolo Valente wrote:
>>> Il giorno 30 mar 2018, alle ore 18:57, Bart Van Assche =
<bart.vanassche@wdc.com> ha scritto:
>>>=20
>>> On Fri, 2018-03-30 at 10:23 +0200, Paolo Valente wrote:
>>>> Still 4.16-rc1, being that the version for which you reported this
>>>> issue in the first place.
>>>=20
>>> A vanilla v4.16-rc1 kernel is not sufficient to run the srp-test =
software
>>> since RDMA/CM support for the SRP target driver is missing from that =
kernel.
>>> That's why I asked you to use the for-next branch from my github =
repository
>>> in a previous e-mail. Anyway, since the necessary patches are now in
>>> linux-next, the srp-test software can also be run against =
linux-next. Here
>>> are the results that I obtained with label next-20180329 and the =
kernel
>>> config attached to your previous e-mail:
>>>=20
>>> # while ./srp-test/run_tests -c -d -r 10 -e bfq; do :; done
>>>=20
>>> BUG: unable to handle kernel NULL pointer dereference at =
0000000000000200
>>> PGD 0 P4D 0=20
>>> Oops: 0002 [#1] SMP PTI
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS =
1.0.0-prebuilt.qemu-project.org 04/01/2014
>>> RIP: 0010:rb_erase+0x284/0x380
>>> Call Trace:
>>> <IRQ>
>>> elv_rb_del+0x24/0x30
>>> bfq_remove_request+0x9a/0x2e0 [bfq]
>>> ? rcu_read_lock_sched_held+0x64/0x70
>>> ? update_load_avg+0x72b/0x760
>>> bfq_finish_requeue_request+0x2e1/0x3b0 [bfq]
>>> ? __lock_is_held+0x5a/0xa0
>>> blk_mq_free_request+0x5f/0x1a0
>>> blk_put_request+0x23/0x60
>>> multipath_release_clone+0xe/0x10
>>> dm_softirq_done+0xe3/0x270
>>> __blk_mq_complete_request_remote+0x18/0x20
>>> flush_smp_call_function_queue+0xa1/0x150
>>> generic_smp_call_function_single_interrupt+0x13/0x30
>>> smp_call_function_single_interrupt+0x4d/0x220
>>> call_function_single_interrupt+0xf/0x20
>>> </IRQ>
>>=20
>> I suspect my recent fix [1] might fix your failure too.
>>=20
>> [1] =
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1682264.html
>=20
> Hello Paolo,
>=20
> With patch [1] applied I can't reproduce the aforementioned crash. I =
will add
> my Tested-by.
>=20

Great, thanks!

Paolo

> Thanks,
>=20
> Bart.
>=20
>=20