Re: NFS-RDMA hangs: connection closed (-103)

[Tom Tucker <tom-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]

Status update...

I have reproduced the bug a number of different ways. It seems to be most 
easily reproduced by simply writing more data than the filesystem has 
space for. I can do this reliably with any FS. I think the XFS bug may 
have tickled this bug somehow.

Tom

On 12/2/10 1:09 PM, Spelic wrote:
> Hello all
> please be aware that the "file oversize" bug is reproducible also 
> without infiniband, with just nfs over ethernet over xfs over ramdisk 
> (but it doesn't hang, so it's a different bug than the one I posted here 
> at the RDMA mailing list)
> I have posted another thread regarding the "file oversize" bug, which 
> you can read in the LVM, XFS, and LKML mailing lists, please have a look
> http://fossplanet.com/f13/%5Blinux-lvm%5D-bugs-mkfs-xfs-device-mapper-xfs-dev-ram-81653/ 
>
> Especially my second post, replying myself at +30 minutes, explains that 
> it's reproducible also with ethernet.
>
> Thank you
>
> On 12/02/2010 07:37 PM, Roland Dreier wrote:
>> Adding Dave Chinner to the cc list, since he's both an XFS guru as well
>> as being very familiar with NFS and RDMA...
>>
>> Dave, if you read below, it seems there is some strange behavior
>> exporting XFS with NFS/RDMA.
>>
>>   - R.
>>
>> >  On 12/02/2010 12:59 AM, Tom Tucker wrote:
>> > >  Spelic,
>> > >
>> > >  I have seen this problem before, but have not been able to reliably
>> > >  reproduce it. When I saw the problem, there were no transport errors
>> > >  and it appeared as if the I/O had actually completed, but that the
>> > >  waiter was not being awoken. I was not able to reliably reproduce
>> > >  the problem and was not able to determine if the problem was a
>> > >  latent bug in NFS in general or a bug in the RDMA transport in
>> > >  particular.
>> > >
>> > >  I will try your setup here, but I don't have a system like yours so
>> > >  I'll have to settle for a smaller ramdisk, however, I have a few
>> > >  questions:
>> > >
>> > >  - Does the FS matter? For example, can you use ext[2-4] on the
>> > >  ramdisk and not still reproduce
>> > >  - As I mentioned earlier NFS v3 vs. NFS v4
>> > >  - RAMDISK size, i.e. 2G vs. 14G
>> > >
>> > >  Thanks,
>> > >  Tom
>> >
>> >  Hello Tom, thanks for replying
>> >
>> >  - The FS matters to some extent: as I wrote, with ext4 it's not
>> >  possible to reproduce the bug in this way, so immediately and
>> >  reliably, however ext4 also will hang eventually if you work on it for
>> >  hours so I had to switch to IPoIB for our real work; reread my
>> >  previous post.
>> >
>> >  - NFS3 not tried yet. Never tried to do RDMA on NFS3... do you have a
>> >  pointer on instructions?
>> >
>> >
>> >  - RAMDISK size: I am testing it.
>> >
>> >  Ok I confirm with 1.5GB ramdisk it's reproducible.
>> >  boot option ramdisk_size=1572864
>> >  (1.5*1024**2=1572864.0)
>> >  confirm: blockdev --getsize64 /dev/ram0 == 1610612736
>> >
>> >  now at server side mkfs and mount with defaults:
>> >  mkfs.xfs /dev/ram0
>> >  mount /dev/ram0 /mnt/ram
>> >  (this is a simplification over my previous email, and it's needed with
>> >  a smaller ramdisk or mkfs.xfs will refuse to work. The bug is still
>> >  reproducible like this)
>> >
>> >
>> >  DOH! another bug:
>> >  It's strange how at the end of the test
>> >      ls -lh /mnt/ram
>> >  at server side will show a zerofile larger than 1.5GB at the end of
>> >  the procedure, sometimes it's 3GB, sometimes it's 2.3GB... but it's
>> >  larger than the ramdisk size.
>> >
>> >  # ll -h /mnt/ram
>> >  total 1.5G
>> >  drwxr-xr-x 2 root root   21 2010-12-02 12:54 ./
>> >  drwxr-xr-x 3 root root 4.0K 2010-11-29 23:51 ../
>> >  -rw-r--r-- 1 root root 2.3G 2010-12-02 12:59 zerofile
>> >  # df -h
>> >  Filesystem            Size  Used Avail Use% Mounted on
>> >  /dev/sda1             294G  4.1G  275G   2% /
>> >  devtmpfs              7.9G  184K  7.9G   1% /dev
>> >  none                  7.9G     0  7.9G   0% /dev/shm
>> >  none                  7.9G  100K  7.9G   1% /var/run
>> >  none                  7.9G     0  7.9G   0% /var/lock
>> >  none                  7.9G     0  7.9G   0% /lib/init/rw
>> >  /dev/ram0             1.5G  1.5G   20K 100% /mnt/ram
>> >
>> >  # dd if=/mnt/ram/zerofile | wc -c
>> >  4791480+0 records in
>> >  4791480+0 records out
>> >  2453237760
>> >  2453237760 bytes (2.5 GB) copied, 8.41821 s, 291 MB/s
>> >
>> >  It seems there is also an XFS bug here...
>> >
>> >  This might help triggering the bug however please note than ext4
>> >  (nfs-rdma over it) also hanged on us and it was real work on HDD disks
>> >  and they were not full... after switching to IPoIB it didn't hang
>> >  anymore.
>> >
>> >  On IPoIB the size problem also shows up: final file is 2.3GB instead
>> >  of<  1.5GB, however nothing hangs:
>> >
>> >  # echo begin; dd if=/dev/zero of=/mnt/nfsram/zerofile bs=1M ; echo
>> >  syncing now ; time sync ; echo finished
>> >  begin
>> >  dd: writing `/mnt/nfsram/zerofile': Input/output error
>> >  2497+0 records in
>> >  2496+0 records out
>> >  2617245696 bytes (2.6 GB) copied, 10.4 s, 252 MB/s
>> >  syncing now
>> >
>> >  real    0m0.057s
>> >  user    0m0.000s
>> >  sys     0m0.000s
>> >  finished
>> >
>> >  I think I noticed the same problem with a 14GB ramdisk, the file ended
>> >  up to be about 15GB, but at that time I thought I made some
>> >  computation mistakes. Now with a smaller ramdisk it's more obvious.
>> >
>> >  Earlier or later someone should notify the XFS developers of the 
>> "size" bug.
>> >  However currently it's a good thing: the size bug might help us to fix
>> >  the RDMA bug.
>> >
>> >  Thanks for your help
>>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html