Re: NFS over RMDA

Re: NFS over RMDA

[poma]

On 25.11.2014 14:29, Josh Boyer wrote:
> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>> Hi,
>>>>
>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>
>>>> I've edited /etc/sysconfig/nfs and set the the following config paramter.
>>>>
>>>> RDMA_PORT=20049
>>>>
>>>> Upon restarting the nfs server I get the following:
>>>>
>>>> modprobe: FATAL: Module svcrdma not found
>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo: 
>>>> write error: Protocol not supported
>>>>
>>>> So it looks like the kernel module svcrdma is missing and the last 
>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to 
>>>> nfs-utils.
>>>>
>>>> Is svcrdma intentionally not built in the current kernels or has it been 
>>>> replaced by something else?
>>>>
>>>
>>> [snip unedited copy from someone's terminal]
>>
>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>> NFSoRDMA client and server support").
>>
>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>> Looking at the git history of the Fedora kernel package I found commit
>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>> choice. Perhaps it was discusses somewhere else.
>>
>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>> either.
>>
>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
> 
> At the time, server support for NFSoRDMA wasn't in the greatest shape
> and we disabled it at the request of the NFS developers.  I believe this
> also matches what wound up in RHEL7.
> 
> The NFS developers haven't asked us to turn it back on, so it has stayed
> off since.
> 
> josh
> 

Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER'?


poma

Re: NFS over RMDA

[Chuck Lever]

On Nov 25, 2014, at 9:26 AM, poma <pomidorabelisima@gmail.com> wrote:

> On 25.11.2014 14:29, Josh Boyer wrote:
>> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>>> Hi,
>>>>> 
>>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>> 
>>>>> I've edited /etc/sysconfig/nfs and set the the following config paramter.
>>>>> 
>>>>> RDMA_PORT=20049
>>>>> 
>>>>> Upon restarting the nfs server I get the following:
>>>>> 
>>>>> modprobe: FATAL: Module svcrdma not found
>>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo: 
>>>>> write error: Protocol not supported
>>>>> 
>>>>> So it looks like the kernel module svcrdma is missing and the last 
>>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to 
>>>>> nfs-utils.
>>>>> 
>>>>> Is svcrdma intentionally not built in the current kernels or has it been 
>>>>> replaced by something else?
>>>>> 
>>>> 
>>>> [snip unedited copy from someone's terminal]
>>> 
>>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>>> NFSoRDMA client and server support").
>>> 
>>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>>> Looking at the git history of the Fedora kernel package I found commit
>>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>>> choice. Perhaps it was discusses somewhere else.
>>> 
>>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>>> either.
>>> 
>>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
>> 
>> At the time, server support for NFSoRDMA wasn't in the greatest shape
>> and we disabled it at the request of the NFS developers.

The NFS/RDMA server crashed often and there was no identified
upstream resource to help with it, so it was surgically
disabled rather than fixed.

>> I believe this
>> also matches what wound up in RHEL7.

>> The NFS developers haven't asked us to turn it back on, so it has stayed
>> off since.
>> 
>> josh
>> 
> 
> Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?

The server in 3.17 and later kernels no longer crashes
regularly.

As to whether you should re-enable it, here’s full
disclosure:

As I understand it the NFS/RDMA server-side transport
effort was defunded after a prototype was merged. It was
hoped that a production-quality implementation would be
funded but it never was.

The code as it stands needs some work, but is operational.
I use it for NFS/RDMA client work every day, so it is
getting some exercise.

Distributions can enable this if they have the resources
(ie, test and support engineers and RDMA adapters) to test
and support customer issues. I assume RH/Fedora do, as
they support client side NFS/RDMA already. But perhaps
consider it as “tech preview” for your enterprise
kernels.

Or they may choose to wait until upstream has changed the
default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.

Note that community support for NFS/RDMA server-side is
still best-effort only. We don’t have any designated
specialists, just a few folks who need the server to work
for other reasons. We are actively looking for interested
parties.

Upstream bugs can be filed here:

  https://bugzilla.linux-nfs.org

HTH

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: NFS over RMDA

[poma]

On 25.11.2014 16:03, Chuck Lever wrote:
> 
> On Nov 25, 2014, at 9:26 AM, poma <pomidorabelisima@gmail.com> wrote:
> 
>> On 25.11.2014 14:29, Josh Boyer wrote:
>>> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>>>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>>>
>>>>>> I've edited /etc/sysconfig/nfs and set the the following config paramter.
>>>>>>
>>>>>> RDMA_PORT=20049
>>>>>>
>>>>>> Upon restarting the nfs server I get the following:
>>>>>>
>>>>>> modprobe: FATAL: Module svcrdma not found
>>>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo: 
>>>>>> write error: Protocol not supported
>>>>>>
>>>>>> So it looks like the kernel module svcrdma is missing and the last 
>>>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to 
>>>>>> nfs-utils.
>>>>>>
>>>>>> Is svcrdma intentionally not built in the current kernels or has it been 
>>>>>> replaced by something else?
>>>>>>
>>>>>
>>>>> [snip unedited copy from someone's terminal]
>>>>
>>>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>>>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>>>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>>>> NFSoRDMA client and server support").
>>>>
>>>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>>>> Looking at the git history of the Fedora kernel package I found commit
>>>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>>>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>>>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>>>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>>>> choice. Perhaps it was discusses somewhere else.
>>>>
>>>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>>>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>>>> either.
>>>>
>>>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>>>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
>>>
>>> At the time, server support for NFSoRDMA wasn't in the greatest shape
>>> and we disabled it at the request of the NFS developers.
> 
> The NFS/RDMA server crashed often and there was no identified
> upstream resource to help with it, so it was surgically
> disabled rather than fixed.
> 
>>> I believe this
>>> also matches what wound up in RHEL7.
> 
>>> The NFS developers haven't asked us to turn it back on, so it has stayed
>>> off since.
>>>
>>> josh
>>>
>>
>> Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
> 
> The server in 3.17 and later kernels no longer crashes
> regularly.
> 
> As to whether you should re-enable it, here’s full
> disclosure:
> 
> As I understand it the NFS/RDMA server-side transport
> effort was defunded after a prototype was merged. It was
> hoped that a production-quality implementation would be
> funded but it never was.
> 
> The code as it stands needs some work, but is operational.
> I use it for NFS/RDMA client work every day, so it is
> getting some exercise.
> 
> Distributions can enable this if they have the resources
> (ie, test and support engineers and RDMA adapters) to test
> and support customer issues. I assume RH/Fedora do, as
> they support client side NFS/RDMA already. But perhaps
> consider it as “tech preview” for your enterprise
> kernels.
> 
> Or they may choose to wait until upstream has changed the
> default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.
> 
> Note that community support for NFS/RDMA server-side is
> still best-effort only. We don’t have any designated
> specialists, just a few folks who need the server to work
> for other reasons. We are actively looking for interested
> parties.
> 
> Upstream bugs can be filed here:
> 
>   https://bugzilla.linux-nfs.org
> 
> HTH
> 
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> 
> 

Thanks man.


poma