Re: CephFS hangs when writing 10GB files in loop

[Wido den Hollander <wido@42on.com>]

On 12/17/2014 05:40 PM, Sage Weil wrote:
> On Wed, 17 Dec 2014, Wido den Hollander wrote:
>> Hi,
>>
>> Today I've been playing with CephFS and the morning started great with
>> CephFS playing along just fine.
>>
>> Some information first:
>> - Ceph 0.89
>> - Linux kernel 3.18
>> - Ceph fuse 0.89
>> - One Active MDS, one Standby
>>
>> This morning I could write a 10GB file like this using the kclient:
>> $ dd if=/dev/zero of=10GB.bin bs=1M count=10240 conv=fsync
>>
>> That gave me 850MB/sec (all 10G network) and I could read the same file
>> again with 610MB/sec.
>>
>> After writing to it multiple times it suddenly started to hang.
>>
>> No real evidence on the MDS (debug mds set to 20) or anything on the
>> client. That specific operation just blocked, but I could still 'ls' the
>> filesystem in a second terminal.
>>
>> The MDS was showing in it's log that it was checking active sessions of
>> clients. It showed the active session of my single client.
>>
>> The client renewed it's caps and proceeded.
>>
>> I currently don't have any logs, but I'm just looking for a direction to
>> be pointed towards.
> 
> Hmm.  Try
> 
>  cat /sys/kernel/debug/ceph/*/mdsc
>  cat /sys/kernel/debug/ceph/*/osdc
> 

I'll check that, good point.

> to see requests in flight (you may need to mount -t debugfs none 
> /sys/kernel/debug first).  What kernel version?
> 

I tried with 3.18

Also tried with ceph-fuse 0.89, same result. It is slower, but it also
hangs at some point.

> sage
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on