Re: NFS-RDMA hangs: connection closed (-103)

[Spelic <spelic-9AbUPqfR1/2XDw4h08c5KA@public.gmane.org>]

Tom, have you reproduced the "RDMA hangs - connection closes" bug or the 
"sparse file at server side upon NFS hitting ENOSPC" ?

Because for the latter people have already given exhaustive explanation: 
see this other thread at 
http://fossplanet.com/f13/%5Blinux-lvm%5D-bugs-mkfs-xfs-device-mapper-xfs-dev-ram-81653/ 


While the former bug is still open and very interesting for us.

Thanks for your help
S.


On 12/07/2010 05:12 PM, Tom Tucker wrote:
> Status update...
>
> I have reproduced the bug a number of different ways. It seems to be 
> most easily reproduced by simply writing more data than the filesystem 
> has space for. I can do this reliably with any FS. I think the XFS bug 
> may have tickled this bug somehow.
>
> Tom
>
> On 12/2/10 1:09 PM, Spelic wrote:
>> Hello all
>> please be aware that the "file oversize" bug is reproducible also 
>> without infiniband, with just nfs over ethernet over xfs over ramdisk 
>> (but it doesn't hang, so it's a different bug than the one I posted 
>> here at the RDMA mailing list)
>> I have posted another thread regarding the "file oversize" bug, which 
>> you can read in the LVM, XFS, and LKML mailing lists, please have a look
>> http://fossplanet.com/f13/%5Blinux-lvm%5D-bugs-mkfs-xfs-device-mapper-xfs-dev-ram-81653/ 
>>
>> Especially my second post, replying myself at +30 minutes, explains 
>> that it's reproducible also with ethernet.
>>
>> Thank you
>>
>> On 12/02/2010 07:37 PM, Roland Dreier wrote:
>>> Adding Dave Chinner to the cc list, since he's both an XFS guru as well
>>> as being very familiar with NFS and RDMA...
>>>
>>> Dave, if you read below, it seems there is some strange behavior
>>> exporting XFS with NFS/RDMA.
>>>
>>>   - R.
>>>
>>> >  On 12/02/2010 12:59 AM, Tom Tucker wrote:
>>> > >  Spelic,
>>> > >
>>> > >  I have seen this problem before, but have not been able to 
>>> reliably
>>> > >  reproduce it. When I saw the problem, there were no transport 
>>> errors
>>> > >  and it appeared as if the I/O had actually completed, but that the
>>> > >  waiter was not being awoken. I was not able to reliably reproduce
>>> > >  the problem and was not able to determine if the problem was a
>>> > >  latent bug in NFS in general or a bug in the RDMA transport in
>>> > >  particular.
>>> > >
>>> > >  I will try your setup here, but I don't have a system like 
>>> yours so
>>> > >  I'll have to settle for a smaller ramdisk, however, I have a few
>>> > >  questions:
>>> > >
>>> > >  - Does the FS matter? For example, can you use ext[2-4] on the
>>> > >  ramdisk and not still reproduce
>>> > >  - As I mentioned earlier NFS v3 vs. NFS v4
>>> > >  - RAMDISK size, i.e. 2G vs. 14G
>>> > >
>>> > >  Thanks,
>>> > >  Tom
>>> >
>>> >  Hello Tom, thanks for replying
>>> >
>>> >  - The FS matters to some extent: as I wrote, with ext4 it's not
>>> >  possible to reproduce the bug in this way, so immediately and
>>> >  reliably, however ext4 also will hang eventually if you work on 
>>> it for
>>> >  hours so I had to switch to IPoIB for our real work; reread my
>>> >  previous post.
>>> >
>>> >  - NFS3 not tried yet. Never tried to do RDMA on NFS3... do you 
>>> have a
>>> >  pointer on instructions?
>>> >
>>> >
>>> >  - RAMDISK size: I am testing it.
>>> >
>>> >  Ok I confirm with 1.5GB ramdisk it's reproducible.
>>> >  boot option ramdisk_size=1572864
>>> >  (1.5*1024**2=1572864.0)
>>> >  confirm: blockdev --getsize64 /dev/ram0 == 1610612736
>>> >
>>> >  now at server side mkfs and mount with defaults:
>>> >  mkfs.xfs /dev/ram0
>>> >  mount /dev/ram0 /mnt/ram
>>> >  (this is a simplification over my previous email, and it's needed 
>>> with
>>> >  a smaller ramdisk or mkfs.xfs will refuse to work. The bug is still
>>> >  reproducible like this)
>>> >
>>> >
>>> >  DOH! another bug:
>>> >  It's strange how at the end of the test
>>> >      ls -lh /mnt/ram
>>> >  at server side will show a zerofile larger than 1.5GB at the end of
>>> >  the procedure, sometimes it's 3GB, sometimes it's 2.3GB... but it's
>>> >  larger than the ramdisk size.
>>> >
>>> >  # ll -h /mnt/ram
>>> >  total 1.5G
>>> >  drwxr-xr-x 2 root root   21 2010-12-02 12:54 ./
>>> >  drwxr-xr-x 3 root root 4.0K 2010-11-29 23:51 ../
>>> >  -rw-r--r-- 1 root root 2.3G 2010-12-02 12:59 zerofile
>>> >  # df -h
>>> >  Filesystem            Size  Used Avail Use% Mounted on
>>> >  /dev/sda1             294G  4.1G  275G   2% /
>>> >  devtmpfs              7.9G  184K  7.9G   1% /dev
>>> >  none                  7.9G     0  7.9G   0% /dev/shm
>>> >  none                  7.9G  100K  7.9G   1% /var/run
>>> >  none                  7.9G     0  7.9G   0% /var/lock
>>> >  none                  7.9G     0  7.9G   0% /lib/init/rw
>>> >  /dev/ram0             1.5G  1.5G   20K 100% /mnt/ram
>>> >
>>> >  # dd if=/mnt/ram/zerofile | wc -c
>>> >  4791480+0 records in
>>> >  4791480+0 records out
>>> >  2453237760
>>> >  2453237760 bytes (2.5 GB) copied, 8.41821 s, 291 MB/s
>>> >
>>> >  It seems there is also an XFS bug here...
>>> >
>>> >  This might help triggering the bug however please note than ext4
>>> >  (nfs-rdma over it) also hanged on us and it was real work on HDD 
>>> disks
>>> >  and they were not full... after switching to IPoIB it didn't hang
>>> >  anymore.
>>> >
>>> >  On IPoIB the size problem also shows up: final file is 2.3GB instead
>>> >  of<  1.5GB, however nothing hangs:
>>> >
>>> >  # echo begin; dd if=/dev/zero of=/mnt/nfsram/zerofile bs=1M ; echo
>>> >  syncing now ; time sync ; echo finished
>>> >  begin
>>> >  dd: writing `/mnt/nfsram/zerofile': Input/output error
>>> >  2497+0 records in
>>> >  2496+0 records out
>>> >  2617245696 bytes (2.6 GB) copied, 10.4 s, 252 MB/s
>>> >  syncing now
>>> >
>>> >  real    0m0.057s
>>> >  user    0m0.000s
>>> >  sys     0m0.000s
>>> >  finished
>>> >
>>> >  I think I noticed the same problem with a 14GB ramdisk, the file 
>>> ended
>>> >  up to be about 15GB, but at that time I thought I made some
>>> >  computation mistakes. Now with a smaller ramdisk it's more obvious.
>>> >
>>> >  Earlier or later someone should notify the XFS developers of the 
>>> "size" bug.
>>> >  However currently it's a good thing: the size bug might help us 
>>> to fix
>>> >  the RDMA bug.
>>> >
>>> >  Thanks for your help
>>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html