Re: NFS-RDMA hangs: connection closed (-103)

[Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>]

Adding Dave Chinner to the cc list, since he's both an XFS guru as well
as being very familiar with NFS and RDMA...

Dave, if you read below, it seems there is some strange behavior
exporting XFS with NFS/RDMA.

 - R.

 > On 12/02/2010 12:59 AM, Tom Tucker wrote:
 > > Spelic,
 > >
 > > I have seen this problem before, but have not been able to reliably
 > > reproduce it. When I saw the problem, there were no transport errors
 > > and it appeared as if the I/O had actually completed, but that the
 > > waiter was not being awoken. I was not able to reliably reproduce
 > > the problem and was not able to determine if the problem was a
 > > latent bug in NFS in general or a bug in the RDMA transport in
 > > particular.
 > >
 > > I will try your setup here, but I don't have a system like yours so
 > > I'll have to settle for a smaller ramdisk, however, I have a few
 > > questions:
 > >
 > > - Does the FS matter? For example, can you use ext[2-4] on the
 > > ramdisk and not still reproduce
 > > - As I mentioned earlier NFS v3 vs. NFS v4
 > > - RAMDISK size, i.e. 2G vs. 14G
 > >
 > > Thanks,
 > > Tom
 > 
 > Hello Tom, thanks for replying
 > 
 > - The FS matters to some extent: as I wrote, with ext4 it's not
 > possible to reproduce the bug in this way, so immediately and
 > reliably, however ext4 also will hang eventually if you work on it for
 > hours so I had to switch to IPoIB for our real work; reread my
 > previous post.
 > 
 > - NFS3 not tried yet. Never tried to do RDMA on NFS3... do you have a
 > pointer on instructions?
 > 
 > 
 > - RAMDISK size: I am testing it.
 > 
 > Ok I confirm with 1.5GB ramdisk it's reproducible.
 > boot option ramdisk_size=1572864
 > (1.5*1024**2=1572864.0)
 > confirm: blockdev --getsize64 /dev/ram0 == 1610612736
 > 
 > now at server side mkfs and mount with defaults:
 > mkfs.xfs /dev/ram0
 > mount /dev/ram0 /mnt/ram
 > (this is a simplification over my previous email, and it's needed with
 > a smaller ramdisk or mkfs.xfs will refuse to work. The bug is still
 > reproducible like this)
 > 
 > 
 > DOH! another bug:
 > It's strange how at the end of the test
 >     ls -lh /mnt/ram
 > at server side will show a zerofile larger than 1.5GB at the end of
 > the procedure, sometimes it's 3GB, sometimes it's 2.3GB... but it's
 > larger than the ramdisk size.
 > 
 > # ll -h /mnt/ram
 > total 1.5G
 > drwxr-xr-x 2 root root   21 2010-12-02 12:54 ./
 > drwxr-xr-x 3 root root 4.0K 2010-11-29 23:51 ../
 > -rw-r--r-- 1 root root 2.3G 2010-12-02 12:59 zerofile
 > # df -h
 > Filesystem            Size  Used Avail Use% Mounted on
 > /dev/sda1             294G  4.1G  275G   2% /
 > devtmpfs              7.9G  184K  7.9G   1% /dev
 > none                  7.9G     0  7.9G   0% /dev/shm
 > none                  7.9G  100K  7.9G   1% /var/run
 > none                  7.9G     0  7.9G   0% /var/lock
 > none                  7.9G     0  7.9G   0% /lib/init/rw
 > /dev/ram0             1.5G  1.5G   20K 100% /mnt/ram
 > 
 > # dd if=/mnt/ram/zerofile | wc -c
 > 4791480+0 records in
 > 4791480+0 records out
 > 2453237760
 > 2453237760 bytes (2.5 GB) copied, 8.41821 s, 291 MB/s
 > 
 > It seems there is also an XFS bug here...
 > 
 > This might help triggering the bug however please note than ext4
 > (nfs-rdma over it) also hanged on us and it was real work on HDD disks
 > and they were not full... after switching to IPoIB it didn't hang
 > anymore.
 > 
 > On IPoIB the size problem also shows up: final file is 2.3GB instead
 > of < 1.5GB, however nothing hangs:
 > 
 > # echo begin; dd if=/dev/zero of=/mnt/nfsram/zerofile bs=1M ; echo
 > syncing now ; time sync ; echo finished
 > begin
 > dd: writing `/mnt/nfsram/zerofile': Input/output error
 > 2497+0 records in
 > 2496+0 records out
 > 2617245696 bytes (2.6 GB) copied, 10.4 s, 252 MB/s
 > syncing now
 > 
 > real    0m0.057s
 > user    0m0.000s
 > sys     0m0.000s
 > finished
 > 
 > I think I noticed the same problem with a 14GB ramdisk, the file ended
 > up to be about 15GB, but at that time I thought I made some
 > computation mistakes. Now with a smaller ramdisk it's more obvious.
 > 
 > Earlier or later someone should notify the XFS developers of the "size" bug.
 > However currently it's a good thing: the size bug might help us to fix
 > the RDMA bug.
 > 
 > Thanks for your help
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html