Re: [PATCH] rfs: Receive Flow Steering

[Changli Gao <xiaosuo@gmail.com>]

On Fri, Apr 2, 2010 at 11:59 AM, Tom Herbert <therbert@google.com> wrote:
> @@ -714,6 +716,8 @@ int inet_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
>  {
>        struct sock *sk = sock->sk;
>
> +       inet_rps_record_flow(sk);
> +
>        /* We may need to bind the socket. */
>        if (!inet_sk(sk)->inet_num && inet_autobind(sk))
>                return -EAGAIN;
> @@ -722,12 +726,13 @@ int inet_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
>  }
>  EXPORT_SYMBOL(inet_sendmsg);
>
> -
>  static ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset,
>                             size_t size, int flags)
>  {
>        struct sock *sk = sock->sk;
>
> +       inet_rps_record_flow(sk);
> +
>        /* We may need to bind the socket. */
>        if (!inet_sk(sk)->inet_num && inet_autobind(sk))
>                return -EAGAIN;
> @@ -737,6 +742,22 @@ static ssize_t inet_sendpage(struct socket *sock, struct page *page, int offset,
>        return sock_no_sendpage(sock, page, offset, size, flags);
>  }
>

for sending packets, how about letting sender compute the rxhash of
the packets from the other side if the rxhash of socket hasn't been
set yet. I is better for client applications.

For router and bridge, the current RPS can work well, but not for
server or client applications. So I propose a new socket option to get
the rps cpu of the packets received on a socket. It may be like this:

int cpu;
getsockopt(sock, SOL_SOCKET, SO_RPSCPU, &cpu, sizeof(cpu));

As Tom's patch did, rxhash is recorded in socket. When the call above
is made, rps_map is looked up to find the RPSCPU for that hash. Once
we get the cpu of the current connection, for a TCP server, it can
dispatch the new connection to the processes which run on that CPU.
the server code will be like this:

fd = accpet(fd, NULL, NULL);
getsockopt(fd, SOL_SOCKET, SO_RPSCPU, &cpu, sizeof(cpu));
asyncq_enqueue(work_queue[cpu], fd);

For a client program, the rxhash can be got after the first packet of
the connection is sent. So the client code will be:

fd = connect(fd, &addr, addr_len);
getsockopt(fd, SOL_SOCKET, SO_RPSCPU, &cpu, sizeof(cpu));
asyncq_enqueue(work_queue[cpu], fd);

I do think this idea is easier to understood. I'll cook a patch later
if it is welcomed.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)