Issues with starting srp_daemon userspace for a 4.13 kernel

Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

Hello

What changed in 4.13 so SRP will no longer start because QP cannot 
modify state.

Same test bed as always (based on RHEL 7.3)

Linux dhcp40-139.desklab.eng.bos.redhat.com 4.12.0

Works and srp_daemon activates using:

run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i 
mlx5_0 -p 1 1>/root/srp1.log 2>&1 &
run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i 
mlx5_1 -p 1 1>/root/srp2.log 2>&1 &

[  158.020366] scsi host2: SRP.T10:7CFE900300726E4E
[  158.574651] scsi host1: SRP.T10:7CFE900300726E4E

Now Testing 4.13, same back-end target that IS running 4.13

Client now booted into 4.13

Linux dhcp40-139.desklab.eng.bos.redhat.com 4.13.0-rc1+

[root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 
-T 10 -t 7000 -ance -i mlx5_0 -p 1
srp_daemon[4573]: failed to modify QP state to RTR
srp_daemon[4573]: failed to modify QP state from RESET to RTS
[root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 
-T 10 -t 7000 -ance -i mlx5_1 -p 1
srp_daemon[4616]: failed to modify QP state to RTR
srp_daemon[4616]: failed to modify QP state from RESET to RTS

Before I start a bisect wanted to ask if I needed a new userspace tools 
for changes in 4.13 maybe

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Jason Gunthorpe]

On Tue, Jul 18, 2017 at 02:07:08PM -0400, Laurence Oberman wrote:
> Hello
> 
> What changed in 4.13 so SRP will no longer start because QP cannot modify
> state.
> 
> Same test bed as always (based on RHEL 7.3)
> 
> Linux dhcp40-139.desklab.eng.bos.redhat.com 4.12.0
> 
> Works and srp_daemon activates using:
> 
> run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i
> mlx5_0 -p 1 1>/root/srp1.log 2>&1 &
> run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i
> mlx5_1 -p 1 1>/root/srp2.log 2>&1 &
> 
> [  158.020366] scsi host2: SRP.T10:7CFE900300726E4E
> [  158.574651] scsi host1: SRP.T10:7CFE900300726E4E
> 
> Now Testing 4.13, same back-end target that IS running 4.13
> 
> Client now booted into 4.13
> 
> Linux dhcp40-139.desklab.eng.bos.redhat.com 4.13.0-rc1+
> 
> [root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10
> -t 7000 -ance -i mlx5_0 -p 1
> srp_daemon[4573]: failed to modify QP state to RTR
> srp_daemon[4573]: failed to modify QP state from RESET to RTS
> [root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10
> -t 7000 -ance -i mlx5_1 -p 1
> srp_daemon[4616]: failed to modify QP state to RTR
> srp_daemon[4616]: failed to modify QP state from RESET to RTS

Do you have this patch?

> Subject: [PATCH v2 1/2] RDMA/uverbs: Fix the check for port number
>
> The port number is only valid if IB_QP_PORT is set in the mask.
> So only check port number if it is valid to prevent modify_qp from
> failing due to an invalid port number.
>
> Fixes: 5ecce4c9b17b("Check port number supplied by user verbs cmds")
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v2.6.14+
> Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Signed-off-by: Mustafa Ismail <mustafa.ismail-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

On 07/18/2017 03:48 PM, Jason Gunthorpe wrote:
> On Tue, Jul 18, 2017 at 02:07:08PM -0400, Laurence Oberman wrote:
>> Hello
>>
>> What changed in 4.13 so SRP will no longer start because QP cannot modify
>> state.
>>
>> Same test bed as always (based on RHEL 7.3)
>>
>> Linux dhcp40-139.desklab.eng.bos.redhat.com 4.12.0
>>
>> Works and srp_daemon activates using:
>>
>> run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i
>> mlx5_0 -p 1 1>/root/srp1.log 2>&1 &
>> run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i
>> mlx5_1 -p 1 1>/root/srp2.log 2>&1 &
>>
>> [  158.020366] scsi host2: SRP.T10:7CFE900300726E4E
>> [  158.574651] scsi host1: SRP.T10:7CFE900300726E4E
>>
>> Now Testing 4.13, same back-end target that IS running 4.13
>>
>> Client now booted into 4.13
>>
>> Linux dhcp40-139.desklab.eng.bos.redhat.com 4.13.0-rc1+
>>
>> [root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10
>> -t 7000 -ance -i mlx5_0 -p 1
>> srp_daemon[4573]: failed to modify QP state to RTR
>> srp_daemon[4573]: failed to modify QP state from RESET to RTS
>> [root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 -T 10
>> -t 7000 -ance -i mlx5_1 -p 1
>> srp_daemon[4616]: failed to modify QP state to RTR
>> srp_daemon[4616]: failed to modify QP state from RESET to RTS
> 
> Do you have this patch?
> 
>> Subject: [PATCH v2 1/2] RDMA/uverbs: Fix the check for port number
>>
>> The port number is only valid if IB_QP_PORT is set in the mask.
>> So only check port number if it is valid to prevent modify_qp from
>> failing due to an invalid port number.
>>
>> Fixes: 5ecce4c9b17b("Check port number supplied by user verbs cmds")
>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v2.6.14+
>> Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>> Signed-off-by: Mustafa Ismail <mustafa.ismail-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Jason,

Yep, That patch seems to be in 4.12 and in 4.13 RC1.
4.12 is working.
I am going to add some debug now as I really dont feel like a bisect 
build unless I first try figure out the differences.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Bart Van Assche]

On Tue, 2017-07-18 at 16:01 -0400, Laurence Oberman wrote:
> Yep, That patch seems to be in 4.12 and in 4.13 RC1.
> 4.12 is working.
> I am going to add some debug now as I really dont feel like a bisect 
> build unless I first try figure out the differences.

Hello Laurence,

Is SELinux enabled on your test setup? If so, can you repeat your test
after having disabled it?

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

On 07/18/2017 04:05 PM, Bart Van Assche wrote:
> On Tue, 2017-07-18 at 16:01 -0400, Laurence Oberman wrote:
>> Yep, That patch seems to be in 4.12 and in 4.13 RC1.
>> 4.12 is working.
>> I am going to add some debug now as I really dont feel like a bisect
>> build unless I first try figure out the differences.
> 
> Hello Laurence,
> 
> Is SELinux enabled on your test setup? If so, can you repeat your test
> after having disabled it?
> 
> Bart.
> 

Hi Bart
Thanks for replying.

Its disabled (at least now it is.) although it should have also been 
disabled when I tested.

I will re-run the test in case.

[root@dhcp40-139 ~]# getenforce
Disabled

Just waiting on a kernel build as I decided to start the bisect as it is 
not that many kernels between v12.x and v13 RC1.

I will say there are lots of code changes in V4.13 showing stuff about 
the RTS in the QP states (Mostly in ROCE though)

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

On 07/18/2017 04:27 PM, Laurence Oberman wrote:
> 
> 
> On 07/18/2017 04:05 PM, Bart Van Assche wrote:
>> On Tue, 2017-07-18 at 16:01 -0400, Laurence Oberman wrote:
>>> Yep, That patch seems to be in 4.12 and in 4.13 RC1.
>>> 4.12 is working.
>>> I am going to add some debug now as I really dont feel like a bisect
>>> build unless I first try figure out the differences.
>>
>> Hello Laurence,
>>
>> Is SELinux enabled on your test setup? If so, can you repeat your test
>> after having disabled it?
>>
>> Bart.
>>
> 
> Hi Bart
> Thanks for replying.
> 
> Its disabled (at least now it is.) although it should have also been 
> disabled when I tested.
> 
> I will re-run the test in case.
> 
> [root@dhcp40-139 ~]# getenforce
> Disabled
> 
> Just waiting on a kernel build as I decided to start the bisect as it is 
> not that many kernels between v12.x and v13 RC1.
> 
> I will say there are lots of code changes in V4.13 showing stuff about 
> the RTS in the QP states (Mostly in ROCE though)
> 
> Thanks
> Laurence

Hi Bart

[root@dhcp40-139 ~]# getenforce
Disabled

[root@dhcp40-139 ~]# uname -a
Linux dhcp40-139.desklab.eng.bos.redhat.com 4.13.0-rc1 #1 SMP Tue Jul 18 
09:15:03 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 
-T 10 -t 7000 -ance -i mlx5_0 -p 1
srp_daemon[14691]: failed to modify QP state to RTR
srp_daemon[14691]: failed to modify QP state from RESET to RTS

OK, will do the bisect now

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

On 07/18/2017 04:59 PM, Laurence Oberman wrote:
> 
> 
> On 07/18/2017 04:27 PM, Laurence Oberman wrote:
>>
>>
>> On 07/18/2017 04:05 PM, Bart Van Assche wrote:
>>> On Tue, 2017-07-18 at 16:01 -0400, Laurence Oberman wrote:
>>>> Yep, That patch seems to be in 4.12 and in 4.13 RC1.
>>>> 4.12 is working.
>>>> I am going to add some debug now as I really dont feel like a bisect
>>>> build unless I first try figure out the differences.
>>>
>>> Hello Laurence,
>>>
>>> Is SELinux enabled on your test setup? If so, can you repeat your test
>>> after having disabled it?
>>>
>>> Bart.
>>>
>>
>> Hi Bart
>> Thanks for replying.
>>
>> Its disabled (at least now it is.) although it should have also been 
>> disabled when I tested.
>>
>> I will re-run the test in case.
>>
>> [root@dhcp40-139 ~]# getenforce
>> Disabled
>>
>> Just waiting on a kernel build as I decided to start the bisect as it 
>> is not that many kernels between v12.x and v13 RC1.
>>
>> I will say there are lots of code changes in V4.13 showing stuff about 
>> the RTS in the QP states (Mostly in ROCE though)
>>
>> Thanks
>> Laurence
> 
> Hi Bart
> 
> [root@dhcp40-139 ~]# getenforce
> Disabled
> 
> [root@dhcp40-139 ~]# uname -a
> Linux dhcp40-139.desklab.eng.bos.redhat.com 4.13.0-rc1 #1 SMP Tue Jul 18 
> 09:15:03 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> 
> [root@dhcp40-139 ~]# run_srp_daemon  -f /etc/ddn/srp_daemon.conf -R 30 
> -T 10 -t 7000 -ance -i mlx5_0 -p 1
> srp_daemon[14691]: failed to modify QP state to RTR
> srp_daemon[14691]: failed to modify QP state from RESET to RTS
> 
> OK, will do the bisect now
> 
> Thanks
> Laurence

So
After many kernels and testing through the bisect I got to the patch 
Jason called out

In other words this is the offender, not the fix, at least for 4.13.

[5ecce4c9b17bed4dc9cb58bfb10447307569b77b] RDMA/uverbs: Check port 
number supplied by user verbs cmds

commit 5ecce4c9b17bed4dc9cb58bfb10447307569b77b
Author: Boris Pismenny <borisp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Date:   Tue Jun 27 15:09:13 2017 +0300

     RDMA/uverbs: Check port number supplied by user verbs cmds

     The ib_uverbs_create_ah() ind ib_uverbs_modify_qp() calls receive
     the port number from user input as part of its attributes and assumes
     it is valid. Down on the stack, that parameter is used to access kernel
     data structures.  If the value is invalid, the kernel accesses memory
     it should not.  To prevent this, verify the port number before 
using it.

     BUG: KASAN: use-after-free in ib_uverbs_create_ah+0x6d5/0x7b0
     Read of size 4 at addr ffff880018d67ab8 by task syz-executor/313

     BUG: KASAN: slab-out-of-bounds in modify_qp.isra.4+0x19d0/0x1ef0
     Read of size 4 at addr ffff88006c40ec58 by task syz-executor/819

     Fixes: 67cdb40ca444 ("[IB] uverbs: Implement more commands")
     Fixes: 189aba99e70 ("IB/uverbs: Extend modify_qp and support packet 
pacing")
     Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v2.6.14+
     Cc: <security-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
     Cc: Yevgeny Kliteynik <kliteyn-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
     Cc: Tziporet Koren <tziporet-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
     Cc: Alex Polak <alexpo-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
     Signed-off-by: Boris Pismenny <borisp-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
     Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
     Signed-off-by: Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>


Jason did I misunderstand you.
That you were saying this patch was the issue.
I thought you were saying this was the fix.

Also this patch is in 4.12.x and 4.12 works so clearly a 4.13.x
compatibility issue.

Apologies if I misunderstood.

If I revert this SRP behaves again on 4.13 RC1

So we will have to revert or fix this patch.
I will see if I can make progress on why it breaks 4.13

Thanks
Laurence




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Jason Gunthorpe]

On Wed, Jul 19, 2017 at 03:40:06PM -0400, Laurence Oberman wrote:

> Jason did I misunderstand you.
> That you were saying this patch was the issue.
> I thought you were saying this was the fix.

Er, that isn't the patch I quoted, that is the broken patch the patch
I quoted fixes:

>> Subject: [PATCH v2 1/2] RDMA/uverbs: Fix the check for port number
>>
>> The port number is only valid if IB_QP_PORT is set in the mask.
>> So only check port number if it is valid to prevent modify_qp from
>> failing due to an invalid port number.
>>
>> Fixes: 5ecce4c9b17b("Check port number supplied by user verbs cmds")
>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v2.6.14+
>> Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>> Signed-off-by: Mustafa Ismail <mustafa.ismail-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

You need the above if you have 5ecce4c9b17b in your tree.

Not sure why it still doesn't seem to be in Linus's tree...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issues with starting srp_daemon userspace for a 4.13 kernel

[Laurence Oberman]

On 07/19/2017 03:49 PM, Jason Gunthorpe wrote:
> On Wed, Jul 19, 2017 at 03:40:06PM -0400, Laurence Oberman wrote:
> 
>> Jason did I misunderstand you.
>> That you were saying this patch was the issue.
>> I thought you were saying this was the fix.
> 
> Er, that isn't the patch I quoted, that is the broken patch the patch
> I quoted fixes:
> 
>>> Subject: [PATCH v2 1/2] RDMA/uverbs: Fix the check for port number
>>>
>>> The port number is only valid if IB_QP_PORT is set in the mask.
>>> So only check port number if it is valid to prevent modify_qp from
>>> failing due to an invalid port number.
>>>
>>> Fixes: 5ecce4c9b17b("Check port number supplied by user verbs cmds")
>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # v2.6.14+
>>> Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>>> Signed-off-by: Mustafa Ismail <mustafa.ismail-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> You need the above if you have 5ecce4c9b17b in your tree.
> 
> Not sure why it still doesn't seem to be in Linus's tree...
> 
> Jason
> 

Groan!!
Apologies,
I grabbed the bad patch commit id when you first responded
5ecce4c9b17b

So that bad patch is in 4.12 and 4.13
Instead of searching for the patch String that fixed it in your original 
message

OK, makes sense now
Nevertheless, needs to go into Linus's tree as you called out.
Thanks
Sorry for the noise
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html