Re: NFS over RMDA

[poma <pomidorabelisima@gmail.com>]

On 25.11.2014 16:03, Chuck Lever wrote:
> 
> On Nov 25, 2014, at 9:26 AM, poma <pomidorabelisima@gmail.com> wrote:
> 
>> On 25.11.2014 14:29, Josh Boyer wrote:
>>> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>>>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>>>
>>>>>> I've edited /etc/sysconfig/nfs and set the the following config paramter.
>>>>>>
>>>>>> RDMA_PORT=20049
>>>>>>
>>>>>> Upon restarting the nfs server I get the following:
>>>>>>
>>>>>> modprobe: FATAL: Module svcrdma not found
>>>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo: 
>>>>>> write error: Protocol not supported
>>>>>>
>>>>>> So it looks like the kernel module svcrdma is missing and the last 
>>>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to 
>>>>>> nfs-utils.
>>>>>>
>>>>>> Is svcrdma intentionally not built in the current kernels or has it been 
>>>>>> replaced by something else?
>>>>>>
>>>>>
>>>>> [snip unedited copy from someone's terminal]
>>>>
>>>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>>>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>>>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>>>> NFSoRDMA client and server support").
>>>>
>>>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>>>> Looking at the git history of the Fedora kernel package I found commit
>>>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>>>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>>>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>>>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>>>> choice. Perhaps it was discusses somewhere else.
>>>>
>>>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>>>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>>>> either.
>>>>
>>>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>>>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
>>>
>>> At the time, server support for NFSoRDMA wasn't in the greatest shape
>>> and we disabled it at the request of the NFS developers.
> 
> The NFS/RDMA server crashed often and there was no identified
> upstream resource to help with it, so it was surgically
> disabled rather than fixed.
> 
>>> I believe this
>>> also matches what wound up in RHEL7.
> 
>>> The NFS developers haven't asked us to turn it back on, so it has stayed
>>> off since.
>>>
>>> josh
>>>
>>
>> Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
> 
> The server in 3.17 and later kernels no longer crashes
> regularly.
> 
> As to whether you should re-enable it, here’s full
> disclosure:
> 
> As I understand it the NFS/RDMA server-side transport
> effort was defunded after a prototype was merged. It was
> hoped that a production-quality implementation would be
> funded but it never was.
> 
> The code as it stands needs some work, but is operational.
> I use it for NFS/RDMA client work every day, so it is
> getting some exercise.
> 
> Distributions can enable this if they have the resources
> (ie, test and support engineers and RDMA adapters) to test
> and support customer issues. I assume RH/Fedora do, as
> they support client side NFS/RDMA already. But perhaps
> consider it as “tech preview” for your enterprise
> kernels.
> 
> Or they may choose to wait until upstream has changed the
> default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.
> 
> Note that community support for NFS/RDMA server-side is
> still best-effort only. We don’t have any designated
> specialists, just a few folks who need the server to work
> for other reasons. We are actively looking for interested
> parties.
> 
> Upstream bugs can be filed here:
> 
>   https://bugzilla.linux-nfs.org
> 
> HTH
> 
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> 
> 

Thanks man.


poma