From: Avi Kivity <avi@redhat.com>
To: "Myklebust, Trond" <Trond.Myklebust@netapp.com>
Cc: Andrew Holway <a.holway@syseleven.de>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"libvir-list@redhat.com" <libvir-list@redhat.com>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: NFS over RDMA small block DIRECT_IO bug
Date: Wed, 05 Sep 2012 17:02:31 +0300 [thread overview]
Message-ID: <50475B77.7090304@redhat.com> (raw)
In-Reply-To: <4FA345DA4F4AE44899BD2B03EEEC2FA908F82C6C@SACEXCMBX04-PRD.hq.netapp.com>
On 09/04/2012 03:04 PM, Myklebust, Trond wrote:
> On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
>> Hello.
>>
>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>>
>> I have a Centos 6.2 server and Centos 6.2 client.
>>
>> [root@store ~]# cat /etc/exports
>> /dev/shm 10.149.0.0/16(rw,fsid=1,no_root_squash,insecure) (I have tried with non tempfs targets also)
>>
>>
>> [root@node001 ~]# cat /etc/fstab
>> store.ibnet:/dev/shm /mnt nfs rdma,port=2050,defaults 0 0
>>
>>
>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>>
>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
>
>
> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
> so that it can use the more efficient RDMA READ and RDMA WRITE memory
> semantics (instead of the SEND/RECEIVE channel semantics).
Shouldn't subpage requests fail then? O_DIRECT block requests fail for
subsector writes, instead of corrupting your data.
Hopefully this is documented somewhere.
--
error compiling committee.c: too many arguments to function
WARNING: multiple messages have this Message-ID (diff)
From: Avi Kivity <avi-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: "Myklebust,
Trond" <Trond.Myklebust-HgOvQuBEEgTQT0dZR+AlfA@public.gmane.org>
Cc: Andrew Holway <a.holway-V6oZj8nR2NRn68oJJulU0Q@public.gmane.org>,
"kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
"libvir-list-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org"
<libvir-list-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
"linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: NFS over RDMA small block DIRECT_IO bug
Date: Wed, 05 Sep 2012 17:02:31 +0300 [thread overview]
Message-ID: <50475B77.7090304@redhat.com> (raw)
In-Reply-To: <4FA345DA4F4AE44899BD2B03EEEC2FA908F82C6C-VGJ86jkg8PeBTCzCEi1xmXOR4cbS7gtM96Bgd4bDwmQ@public.gmane.org>
On 09/04/2012 03:04 PM, Myklebust, Trond wrote:
> On Tue, 2012-09-04 at 11:31 +0200, Andrew Holway wrote:
>> Hello.
>>
>> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
>>
>> I have a Centos 6.2 server and Centos 6.2 client.
>>
>> [root@store ~]# cat /etc/exports
>> /dev/shm 10.149.0.0/16(rw,fsid=1,no_root_squash,insecure) (I have tried with non tempfs targets also)
>>
>>
>> [root@node001 ~]# cat /etc/fstab
>> store.ibnet:/dev/shm /mnt nfs rdma,port=2050,defaults 0 0
>>
>>
>> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
>>
>> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
>
>
> That is expected behaviour. DIRECT_IO over RDMA needs to be page aligned
> so that it can use the more efficient RDMA READ and RDMA WRITE memory
> semantics (instead of the SEND/RECEIVE channel semantics).
Shouldn't subpage requests fail then? O_DIRECT block requests fail for
subsector writes, instead of corrupting your data.
Hopefully this is documented somewhere.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-09-05 14:02 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-04 9:31 NFS over RDMA small block DIRECT_IO bug Andrew Holway
2012-09-04 12:04 ` Myklebust, Trond
2012-09-04 12:04 ` Myklebust, Trond
2012-09-04 12:52 ` Andrew Holway
2012-09-05 14:02 ` Avi Kivity [this message]
2012-09-05 14:02 ` Avi Kivity
2012-09-06 10:14 ` Andrew Holway
2012-09-06 13:45 ` Myklebust, Trond
2012-09-06 13:45 ` Myklebust, Trond
2012-09-11 17:03 ` Steve Dickson
2012-09-11 17:03 ` Steve Dickson
2012-09-18 14:03 ` Andrew Holway
2012-09-18 14:03 ` Andrew Holway
2012-09-19 15:50 ` Steve Dickson
2012-09-19 15:50 ` Steve Dickson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50475B77.7090304@redhat.com \
--to=avi@redhat.com \
--cc=Trond.Myklebust@netapp.com \
--cc=a.holway@syseleven.de \
--cc=kvm@vger.kernel.org \
--cc=libvir-list@redhat.com \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.