Netlink messages without NLM_F_REQUEST flag

Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Hi Kaike,

In the commit bc10ed7d3d19 ("IB/core: Add rdma netlink helper functions"),
part of larger series [1], you introduced ibnl_rcv_reply_skb(), which is very
similar to netlink_rcv_skb() with one major change.

The netlink messages without NLM_F_REQUEST flag are handled by ibnl_rcv_reply_skb(),
while netlink_rcv_skb() doesn't. The comment introduced in commit d35b685640ae
"[NETLINK]: Ignore !NLM_F_REQUEST messages directly in netlink_run_queue()")
says that "Only requests are handled by the kernel".

It makes me wonder if it is expected behavior for ibnl_rcv_reply_skb()
to handle !NLM_F_REQUEST messages and do we really need it? What are the scenarios?
In my use case, which is for sure different from yours, I'm always setting NLM_F_REQUEST
while communicating with kernel.

Thanks

[1] http://www.spinics.net/lists/linux-rdma/msg28153.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Netlink messages without NLM_F_REQUEST flag

[Jason Gunthorpe]

On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> It makes me wonder if it is expected behavior for
> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
> really need it? What are the scenarios?  In my use case, which is
> for sure different from yours, I'm always setting NLM_F_REQUEST
> while communicating with kernel.

If I recall the user space SA code issues REQUESTS from the kernel to
userspace, so userspace returns with the response format. This is
abnormal for netlink hence the special function.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
>> It makes me wonder if it is expected behavior for
>> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
>> really need it? What are the scenarios?  In my use case, which is
>> for sure different from yours, I'm always setting NLM_F_REQUEST
>> while communicating with kernel.
>
> If I recall the user space SA code issues REQUESTS from the kernel to
> userspace, so userspace returns with the response format. This is
> abnormal for netlink hence the special function.

In netlink semantics, kernel side is supposed to send netlink
notification message and userspace is supposed to send REQUEST.

>
> Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Jason Gunthorpe]

On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> >> It makes me wonder if it is expected behavior for
> >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
> >> really need it? What are the scenarios?  In my use case, which is
> >> for sure different from yours, I'm always setting NLM_F_REQUEST
> >> while communicating with kernel.
> >
> > If I recall the user space SA code issues REQUESTS from the kernel to
> > userspace, so userspace returns with the response format. This is
> > abnormal for netlink hence the special function.
> 
> In netlink semantics, kernel side is supposed to send netlink
> notification message and userspace is supposed to send REQUEST.

That pattern is for async communications, the SA stuff needs a sync
protocol, unfortunately.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Wed, Jun 07, 2017 at 10:47:50AM -0600, Jason Gunthorpe wrote:
> On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> > On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
> > <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> > >> It makes me wonder if it is expected behavior for
> > >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
> > >> really need it? What are the scenarios?  In my use case, which is
> > >> for sure different from yours, I'm always setting NLM_F_REQUEST
> > >> while communicating with kernel.
> > >
> > > If I recall the user space SA code issues REQUESTS from the kernel to
> > > userspace, so userspace returns with the response format. This is
> > > abnormal for netlink hence the special function.
> >
> > In netlink semantics, kernel side is supposed to send netlink
> > notification message and userspace is supposed to send REQUEST.
>
> That pattern is for async communications, the SA stuff needs a sync
> protocol, unfortunately.

There is special flag NLM_F_ACK for it and userspace will set
NLM_F_REQUEST | NLM_F_ACK once synchronization is needed.

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Netlink messages without NLM_F_REQUEST flag

[Jason Gunthorpe]

On Wed, Jun 07, 2017 at 08:00:37PM +0300, Leon Romanovsky wrote:
> On Wed, Jun 07, 2017 at 10:47:50AM -0600, Jason Gunthorpe wrote:
> > On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
> > > <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > > > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> > > >> It makes me wonder if it is expected behavior for
> > > >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
> > > >> really need it? What are the scenarios?  In my use case, which is
> > > >> for sure different from yours, I'm always setting NLM_F_REQUEST
> > > >> while communicating with kernel.
> > > >
> > > > If I recall the user space SA code issues REQUESTS from the kernel to
> > > > userspace, so userspace returns with the response format. This is
> > > > abnormal for netlink hence the special function.
> > >
> > > In netlink semantics, kernel side is supposed to send netlink
> > > notification message and userspace is supposed to send REQUEST.
> >
> > That pattern is for async communications, the SA stuff needs a sync
> > protocol, unfortunately.
> 
> There is special flag NLM_F_ACK for it and userspace will set
> NLM_F_REQUEST | NLM_F_ACK once synchronization is needed.

AFAIK, that is different, that is acking and retriggering a single shot
notification, not completing a kernel initiated handshake.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

On Wed, Jun 07, 2017 at 11:07:02AM -0600, Jason Gunthorpe wrote:
> On Wed, Jun 07, 2017 at 08:00:37PM +0300, Leon Romanovsky wrote:
> > On Wed, Jun 07, 2017 at 10:47:50AM -0600, Jason Gunthorpe wrote:
> > > On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
> > > > <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > > > > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> > > > >> It makes me wonder if it is expected behavior for
> > > > >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do we
> > > > >> really need it? What are the scenarios?  In my use case, which is
> > > > >> for sure different from yours, I'm always setting NLM_F_REQUEST
> > > > >> while communicating with kernel.
> > > > >
> > > > > If I recall the user space SA code issues REQUESTS from the kernel to
> > > > > userspace, so userspace returns with the response format. This is
> > > > > abnormal for netlink hence the special function.
> > > >
> > > > In netlink semantics, kernel side is supposed to send netlink
> > > > notification message and userspace is supposed to send REQUEST.
> > >
> > > That pattern is for async communications, the SA stuff needs a sync
> > > protocol, unfortunately.
> >
> > There is special flag NLM_F_ACK for it and userspace will set
> > NLM_F_REQUEST | NLM_F_ACK once synchronization is needed.
>
> AFAIK, that is different, that is acking and retriggering a single shot
> notification, not completing a kernel initiated handshake.

It is acking that message from user was received by kernel and now
processing. The message from kernel to user are anyway unreliable [1],
so I don't understand on which handshake you are talking.

[1] "reliable transmissions from kernel to user are impossible in any case"
https://linux.die.net/man/7/netlink

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Netlink messages without NLM_F_REQUEST flag

[Hefty, Sean]

> It is acking that message from user was received by kernel and now
> processing. The message from kernel to user are anyway unreliable [1],
> so I don't understand on which handshake you are talking.
> 
> [1] "reliable transmissions from kernel to user are impossible in any
> case"
> https://linux.die.net/man/7/netlink

The SA messages are kernel initiated messages that target a user space daemon (e.g. ibacm) for cached PR data.  If that fails because the daemon isn't running or the message is lost, the normal retry mechanism in the kernel should kick in and try sending the SA message to the remote (i.e. real) SA.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Jason Gunthorpe]

On Wed, Jun 07, 2017 at 09:18:10PM +0300, Leon Romanovsky wrote:

> > AFAIK, that is different, that is acking and retriggering a single shot
> > notification, not completing a kernel initiated handshake.
> 
> It is acking that message from user was received by kernel and now
> processing.

But isn't what is cared about here - the SA thing needs to send a
request to user space and collect a reply, it runs the protocol
backwards from normal.

It is not a notification because the request actually needs a reply,
and ACK's don't help because the reply has content.

It does not use the NOTIFICATION/REQUEST/REPLY sequence because it
does not care about reliability of delivering the REQUEST to user
space.

Jason

Re: Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Wed, Jun 07, 2017 at 12:35:11PM -0600, Jason Gunthorpe wrote:
> On Wed, Jun 07, 2017 at 09:18:10PM +0300, Leon Romanovsky wrote:
>
> > > AFAIK, that is different, that is acking and retriggering a single shot
> > > notification, not completing a kernel initiated handshake.
> >
> > It is acking that message from user was received by kernel and now
> > processing.
>
> But isn't what is cared about here - the SA thing needs to send a
> request to user space and collect a reply, it runs the protocol
> backwards from normal.

Thanks all, this explain a little bit "RDMA anomaly".
I'm not sure that I saw such scheme in the kernel, does anybody else
in the kernel use such backward scheme or this is RDMA-specific?

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Netlink messages without NLM_F_REQUEST flag

[Weiny, Ira]

> 
> On Wed, Jun 07, 2017 at 10:47:50AM -0600, Jason Gunthorpe wrote:
> > On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe 
> > > <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > > > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> > > >> It makes me wonder if it is expected behavior for
> > > >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do 
> > > >> we really need it? What are the scenarios?  In my use case, 
> > > >> which is for sure different from yours, I'm always setting 
> > > >> NLM_F_REQUEST while communicating with kernel.
> > > >
> > > > If I recall the user space SA code issues REQUESTS from the 
> > > > kernel to userspace, so userspace returns with the response 
> > > > format. This is abnormal for netlink hence the special function.
> > >
> > > In netlink semantics, kernel side is supposed to send netlink 
> > > notification message and userspace is supposed to send REQUEST.
> >
> > That pattern is for async communications, the SA stuff needs a sync 
> > protocol, unfortunately.
> 
> There is special flag NLM_F_ACK for it and userspace will set 
> NLM_F_REQUEST | NLM_F_ACK once synchronization is needed.
> 

Reference?

>From my understanding, NLM_F_REQUEST | NLM_F_ACK is simply requesting an ack from the kernel on a request.  In our case the message is a response to the kernel request.

Ira


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netlink messages without NLM_F_REQUEST flag

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2063 bytes --]

On Wed, Jun 07, 2017 at 05:10:35PM +0000, Weiny, Ira wrote:
> >
> > On Wed, Jun 07, 2017 at 10:47:50AM -0600, Jason Gunthorpe wrote:
> > > On Wed, Jun 07, 2017 at 07:43:44PM +0300, Leon Romanovsky wrote:
> > > > On Wed, Jun 7, 2017 at 7:37 PM, Jason Gunthorpe
> > > > <jgunthorpe@obsidianresearch.com> wrote:
> > > > > On Wed, Jun 07, 2017 at 07:19:01PM +0300, Leon Romanovsky wrote:
> > > > >> It makes me wonder if it is expected behavior for
> > > > >> ibnl_rcv_reply_skb() to handle !NLM_F_REQUEST messages and do
> > > > >> we really need it? What are the scenarios?  In my use case,
> > > > >> which is for sure different from yours, I'm always setting
> > > > >> NLM_F_REQUEST while communicating with kernel.
> > > > >
> > > > > If I recall the user space SA code issues REQUESTS from the
> > > > > kernel to userspace, so userspace returns with the response
> > > > > format. This is abnormal for netlink hence the special function.
> > > >
> > > > In netlink semantics, kernel side is supposed to send netlink
> > > > notification message and userspace is supposed to send REQUEST.
> > >
> > > That pattern is for async communications, the SA stuff needs a sync
> > > protocol, unfortunately.
> >
> > There is special flag NLM_F_ACK for it and userspace will set
> > NLM_F_REQUEST | NLM_F_ACK once synchronization is needed.
> >
>
> Reference?
>
> From my understanding, NLM_F_REQUEST | NLM_F_ACK is simply requesting an ack from the kernel on a request.  In our case the message is a response to the kernel request.

There is a large chance that we are talking about different aspects of
"sync protocol" and for sure it is due to  my misunderstanding. Can you
elaborate more about the protocol? Why is so unique to RDMA?

Why is ACK + sequence number tracking not enough?

https://www.infradead.org/~tgr/libnl/doc/core.html#core_msg_ack

>
> Ira
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]