ceph_test_librbd_fsx fails with thrasher

ceph_test_librbd_fsx fails with thrasher

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Hi Josh,

While looking at errors from a giant (plus backports) run of the RBD suite, I stumbled upon:

   http://tracker.ceph.com/issues/10513

and the error is unclear (at least to me ;-). Would you have an idea ? For information the backports included are all the pull requests mentionned at http://tracker.ceph.com/issues/10501.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: ceph_test_librbd_fsx fails with thrasher

[Josh Durgin]

On 01/12/2015 06:26 AM, Loic Dachary wrote:
> Hi Josh,
>
> While looking at errors from a giant (plus backports) run of the RBD suite, I stumbled upon:
>
>     http://tracker.ceph.com/issues/10513
>
> and the error is unclear (at least to me ;-). Would you have an idea ? For information the backports included are all the pull requests mentionned at http://tracker.ceph.com/issues/10501.

Thanks for organizing these Loic!

There's no trace of the actual error in the teuthology or ceph client
logs. Looking at the syslogs, this and several other failures from that
run were from running out of memory. This happened with and without
caching, so I suspect these tests may simply need more memory in their
current configuration, since osds are sharing it too, and they use much
more with thrashing.

Josh

Re: ceph_test_librbd_fsx fails with thrasher

[Ilya Dryomov]

On Mon, Jan 12, 2015 at 7:59 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 01/12/2015 06:26 AM, Loic Dachary wrote:
>>
>> Hi Josh,
>>
>> While looking at errors from a giant (plus backports) run of the RBD
>> suite, I stumbled upon:
>>
>>     http://tracker.ceph.com/issues/10513
>>
>> and the error is unclear (at least to me ;-). Would you have an idea ? For
>> information the backports included are all the pull requests mentionned at
>> http://tracker.ceph.com/issues/10501.
>
>
> Thanks for organizing these Loic!
>
> There's no trace of the actual error in the teuthology or ceph client
> logs. Looking at the syslogs, this and several other failures from that
> run were from running out of memory. This happened with and without
> caching, so I suspect these tests may simply need more memory in their
> current configuration, since osds are sharing it too, and they use much
> more with thrashing.

Speaking of fsx and thrasher.  Last time I tried fsx (both kernel and
librbd) couldn't survive thrashing due to watch/notify troubles.  Are
all the new watch/notify pieces that were supposed to fix this in?

Thanks,

                Ilya

Re: ceph_test_librbd_fsx fails with thrasher

[Josh Durgin]

On 01/12/2015 09:13 AM, Ilya Dryomov wrote:
> Speaking of fsx and thrasher.  Last time I tried fsx (both kernel and
> librbd) couldn't survive thrashing due to watch/notify troubles.  Are
> all the new watch/notify pieces that were supposed to fix this in?

There's just one bug left afaik: https://github.com/ceph/ceph/pull/3281