NFS over RDMA small block DIRECT_IO bug

NFS over RDMA small block DIRECT_IO bug

[Andrew Holway]

Hello.

# Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #

I have a Centos 6.2 server and Centos 6.2 client.

[root@store ~]# cat /etc/exports 
/dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)


[root@node001 ~]# cat /etc/fstab 
store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0


I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.

Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.

I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.

Thanks,

Andrew.

bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228

[root@node001 mnt]# for f in 512 1024 2048 4096 8192 16384 32768 65536 131072; do dd bs="$f" if=CentOS-6.3-x86_64-netinstall.iso of=hello iflag=direct oflag=direct && md5sum hello && rm -f hello; done

409600+0 records in
409600+0 records out
209715200 bytes (210 MB) copied, 62.3649 s, 3.4 MB/s
aadd0ffe3c9dfa35d8354e99ecac9276  hello -- 512 byte block 

204800+0 records in
204800+0 records out
209715200 bytes (210 MB) copied, 41.3876 s, 5.1 MB/s
336f6da78f93dab591edc18da81f002e  hello -- 1K block

102400+0 records in
102400+0 records out
209715200 bytes (210 MB) copied, 21.1712 s, 9.9 MB/s
f4cefe0a05c9b47ba68effdb17dc95d6  hello -- 2k block

51200+0 records in
51200+0 records out
209715200 bytes (210 MB) copied, 10.9631 s, 19.1 MB/s
690138908de516b6e5d7d180d085c3f3  hello -- 4k block

25600+0 records in
25600+0 records out
209715200 bytes (210 MB) copied, 5.4136 s, 38.7 MB/s
690138908de516b6e5d7d180d085c3f3  hello

12800+0 records in
12800+0 records out
209715200 bytes (210 MB) copied, 3.1448 s, 66.7 MB/s
690138908de516b6e5d7d180d085c3f3  hello

6400+0 records in
6400+0 records out
209715200 bytes (210 MB) copied, 1.77304 s, 118 MB/s
690138908de516b6e5d7d180d085c3f3  hello

3200+0 records in
3200+0 records out
209715200 bytes (210 MB) copied, 1.4331 s, 146 MB/s
690138908de516b6e5d7d180d085c3f3  hello

1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 0.922167 s, 227 MB/s
690138908de516b6e5d7d180d085c3f3  hello

Re: NFS over RDMA small block DIRECT_IO bug

[Myklebust, Trond]

On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
> Hello.
> 
> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
> 
> I have a Centos 6.2 server and Centos 6.2 client.
> 
> [root@store ~]# cat /etc/exports 
> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
> 
> 
> [root@node001 ~]# cat /etc/fstab 
> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
> 
> 
> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
> 
> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.


That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
so that it can use the more efficient RDMA READ and RDMA WRITE memory
semantics (instead of the SEND/RECEIVE channel semantics).

> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
> 
> Thanks,
> 
> Andrew.
> 
> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
> 
> [root@node001 mnt]# for f in 512 1024 2048 4096 8192 16384 32768 65536 131072; do dd bs="$f" if=CentOS-6.3-x86_64-netinstall.iso of=hello iflag=direct oflag=direct && md5sum hello && rm -f hello; done
> 
> 409600+0 records in
> 409600+0 records out
> 209715200 bytes (210 MB) copied, 62.3649 s, 3.4 MB/s
> aadd0ffe3c9dfa35d8354e99ecac9276  hello -- 512 byte block 
> 
> 204800+0 records in
> 204800+0 records out
> 209715200 bytes (210 MB) copied, 41.3876 s, 5.1 MB/s
> 336f6da78f93dab591edc18da81f002e  hello -- 1K block
> 
> 102400+0 records in
> 102400+0 records out
> 209715200 bytes (210 MB) copied, 21.1712 s, 9.9 MB/s
> f4cefe0a05c9b47ba68effdb17dc95d6  hello -- 2k block
> 
> 51200+0 records in
> 51200+0 records out
> 209715200 bytes (210 MB) copied, 10.9631 s, 19.1 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello -- 4k block
> 
> 25600+0 records in
> 25600+0 records out
> 209715200 bytes (210 MB) copied, 5.4136 s, 38.7 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello
> 
> 12800+0 records in
> 12800+0 records out
> 209715200 bytes (210 MB) copied, 3.1448 s, 66.7 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello
> 
> 6400+0 records in
> 6400+0 records out
> 209715200 bytes (210 MB) copied, 1.77304 s, 118 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello
> 
> 3200+0 records in
> 3200+0 records out
> 209715200 bytes (210 MB) copied, 1.4331 s, 146 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello
> 
> 1600+0 records in
> 1600+0 records out
> 209715200 bytes (210 MB) copied, 0.922167 s, 227 MB/s
> 690138908de516b6e5d7d180d085c3f3  hello
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFS over RDMA small block DIRECT_IO bug

[Andrew Holway]

> 
> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
> so that it can use the more efficient RDMA READ and RDMA WRITE memory
> semantics (instead of the SEND/RECEIVE channel semantics).

Yes, I think I am understanding that now.

I need to find a way of getting around the lib-virt issue.

http://lists.gnu.org/archive/html/qemu-devel/2011-12/msg01570.html

Thanks,

Andrew


> 
>> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
>> 
>> Thanks,
>> 
>> Andrew.
>> 
>> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
>> 
>> [root@node001 mnt]# for f in 512 1024 2048 4096 8192 16384 32768 65536 131072; do dd bs="$f" if=CentOS-6.3-x86_64-netinstall.iso of=hello iflag=direct oflag=direct && md5sum hello && rm -f hello; done
>> 
>> 409600+0 records in
>> 409600+0 records out
>> 209715200 bytes (210 MB) copied, 62.3649 s, 3.4 MB/s
>> aadd0ffe3c9dfa35d8354e99ecac9276  hello -- 512 byte block 
>> 
>> 204800+0 records in
>> 204800+0 records out
>> 209715200 bytes (210 MB) copied, 41.3876 s, 5.1 MB/s
>> 336f6da78f93dab591edc18da81f002e  hello -- 1K block
>> 
>> 102400+0 records in
>> 102400+0 records out
>> 209715200 bytes (210 MB) copied, 21.1712 s, 9.9 MB/s
>> f4cefe0a05c9b47ba68effdb17dc95d6  hello -- 2k block
>> 
>> 51200+0 records in
>> 51200+0 records out
>> 209715200 bytes (210 MB) copied, 10.9631 s, 19.1 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello -- 4k block
>> 
>> 25600+0 records in
>> 25600+0 records out
>> 209715200 bytes (210 MB) copied, 5.4136 s, 38.7 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello
>> 
>> 12800+0 records in
>> 12800+0 records out
>> 209715200 bytes (210 MB) copied, 3.1448 s, 66.7 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello
>> 
>> 6400+0 records in
>> 6400+0 records out
>> 209715200 bytes (210 MB) copied, 1.77304 s, 118 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello
>> 
>> 3200+0 records in
>> 3200+0 records out
>> 209715200 bytes (210 MB) copied, 1.4331 s, 146 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello
>> 
>> 1600+0 records in
>> 1600+0 records out
>> 209715200 bytes (210 MB) copied, 0.922167 s, 227 MB/s
>> 690138908de516b6e5d7d180d085c3f3  hello
>> 
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: NFS over RDMA small block DIRECT_IO bug

[Avi Kivity]

On 09/04/2012 03:04 PM, Myklebust, Trond wrote:
> On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
>> Hello.
>> 
>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>> 
>> I have a Centos 6.2 server and Centos 6.2 client.
>> 
>> [root@store ~]# cat /etc/exports 
>> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
>> 
>> 
>> [root@node001 ~]# cat /etc/fstab 
>> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
>> 
>> 
>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>> 
>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
> 
> 
> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
> so that it can use the more efficient RDMA READ and RDMA WRITE memory
> semantics (instead of the SEND/RECEIVE channel semantics).

Shouldn't subpage requests fail then?  O_DIRECT block requests fail for
subsector writes, instead of corrupting your data.

Hopefully this is documented somewhere.

-- 
error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS over RDMA small block DIRECT_IO bug

[Andrew Holway]

On Sep 5, 2012, at 4:02 PM, Avi Kivity wrote:

> On 09/04/2012 03:04 PM, Myklebust, Trond wrote:
>> On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
>>> Hello.
>>> 
>>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>>> 
>>> I have a Centos 6.2 server and Centos 6.2 client.
>>> 
>>> [root@store ~]# cat /etc/exports 
>>> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
>>> 
>>> 
>>> [root@node001 ~]# cat /etc/fstab 
>>> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
>>> 
>>> 
>>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>>> 
>>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
>> 
>> 
>> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
>> so that it can use the more efficient RDMA READ and RDMA WRITE memory
>> semantics (instead of the SEND/RECEIVE channel semantics).
> 
> Shouldn't subpage requests fail then?  O_DIRECT block requests fail for
> subsector writes, instead of corrupting your data.

But silent data corruption is so much fun!!

Re: NFS over RDMA small block DIRECT_IO bug

[Myklebust, Trond]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1824 bytes --]

On Thu, 2012-09-06 at 12:14 +0200, Andrew Holway wrote:
> On Sep 5, 2012, at 4:02 PM, Avi Kivity wrote:
> 
> > On 09/04/2012 03:04 PM, Myklebust, Trond wrote:
> >> On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
> >>> Hello.
> >>> 
> >>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
> >>> 
> >>> I have a Centos 6.2 server and Centos 6.2 client.
> >>> 
> >>> [root@store ~]# cat /etc/exports 
> >>> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
> >>> 
> >>> 
> >>> [root@node001 ~]# cat /etc/fstab 
> >>> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
> >>> 
> >>> 
> >>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
> >>> 
> >>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
> >> 
> >> 
> >> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
> >> so that it can use the more efficient RDMA READ and RDMA WRITE memory
> >> semantics (instead of the SEND/RECEIVE channel semantics).
> > 
> > Shouldn't subpage requests fail then?  O_DIRECT block requests fail for
> > subsector writes, instead of corrupting your data.
> 
> But silent data corruption is so much fun!!

A couple of RDMA folks are looking into why this is happening. I'm
hoping they will get back to me soon.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±û"žØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&¢îý»\x05ËÛÔØï¦v¬Îf\x1dp)¹¹br	šê+€Ê+zf£¢·hšˆ§~†­†Ûiÿûàz¹\x1e®w¥¢¸?™¨è­Ú&¢)ߢ^[f

Re: NFS over RDMA small block DIRECT_IO bug

[Steve Dickson]

On 09/04/2012 05:31 AM, Andrew Holway wrote:
> Hello.
> 
> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
> 
> I have a Centos 6.2 server and Centos 6.2 client.
> 
> [root@store ~]# cat /etc/exports 
> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
> 
> 
> [root@node001 ~]# cat /etc/fstab 
> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
> 
> 
> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
> 
> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
> 
> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
> 
> Thanks,
> 
> Andrew.
> 
> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
Well it appears the RHEL6 kernels are lacking a couple patches that might
help with this....

5c635e09 RPCRDMA: Fix FRMR registration/invalidate handling.
9b78145c xprtrdma: Remove assumption that each segment is <= PAGE_SIZE

I can only image that Centos 6.2 might me lacking these too... ;-)

steved.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS over RDMA small block DIRECT_IO bug

[Andrew Holway]

Hi Steve,

Do you think these patches will make their way into the redhat kernel sometime soon?

What is the state of support for NFS over RDMA support at redhat?

Thanks,

Andrew


On Sep 11, 2012, at 7:03 PM, Steve Dickson wrote:

> 
> 
> On 09/04/2012 05:31 AM, Andrew Holway wrote:
>> Hello.
>> 
>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>> 
>> I have a Centos 6.2 server and Centos 6.2 client.
>> 
>> [root@store ~]# cat /etc/exports 
>> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
>> 
>> 
>> [root@node001 ~]# cat /etc/fstab 
>> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
>> 
>> 
>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>> 
>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
>> 
>> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
>> 
>> Thanks,
>> 
>> Andrew.
>> 
>> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
> Well it appears the RHEL6 kernels are lacking a couple patches that might
> help with this....
> 
> 5c635e09 RPCRDMA: Fix FRMR registration/invalidate handling.
> 9b78145c xprtrdma: Remove assumption that each segment is <= PAGE_SIZE
> 
> I can only image that Centos 6.2 might me lacking these too... ;-)
> 
> steved.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS over RDMA small block DIRECT_IO bug

[Steve Dickson]

On 09/18/2012 10:03 AM, Andrew Holway wrote:
> Hi Steve,
> 
> Do you think these patches will make their way into the redhat kernel sometime soon?
The process would start by opening a bz at bugzilla.redhat.com... If you like, you
can send me the pointer to the bz and I'll make sure it gets noticed... 
> 
> What is the state of support for NFS over RDMA support at redhat?
In theory its supported, but in reality that post is currently unnamed 
which seems to be the case in upstream as well... 

steved.

> 
> Thanks,
> 
> Andrew
> 
> 
> On Sep 11, 2012, at 7:03 PM, Steve Dickson wrote:
> 
>>
>>
>> On 09/04/2012 05:31 AM, Andrew Holway wrote:
>>> Hello.
>>>
>>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>>>
>>> I have a Centos 6.2 server and Centos 6.2 client.
>>>
>>> [root@store ~]# cat /etc/exports 
>>> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
>>>
>>>
>>> [root@node001 ~]# cat /etc/fstab 
>>> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
>>>
>>>
>>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>>>
>>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
>>>
>>> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
>>>
>>> Thanks,
>>>
>>> Andrew.
>>>
>>> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
>> Well it appears the RHEL6 kernels are lacking a couple patches that might
>> help with this....
>>
>> 5c635e09 RPCRDMA: Fix FRMR registration/invalidate handling.
>> 9b78145c xprtrdma: Remove assumption that each segment is <= PAGE_SIZE
>>
>> I can only image that Centos 6.2 might me lacking these too... ;-)
>>
>> steved.
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html