LTTng tracing: hitting the message throttle

LTTng tracing: hitting the message throttle

[Andreas Bluemle]

Hi,

while running tests to collect data using LTTng, I was hitting
the message throttle in ceph-osd, which is controlled by configuration
option osd_map_message_max, default 100.

If I am not mistaken, then this throttle hits when more than
100 messages received via SimpleMessenger are not yet destructed,
i.e. are still under processing within the affected ceph-osd daemon.

I was running a fio test, random writes of 4 KByte with 16 parallel
IO's. The storage cluster consists of 12 osds on 3 storage nodes,
replication level 3.

I wonder why I hit that message throttle with my load profile:
the 16 parallel IO's should at most generate 48 data messages and also
48 acknowledges across the cluster - so I would not expect anything
even close to a 100 message limit on a single osd in my cluster.

Did anybody else experience such issues?

Are there other throttle values which I should look at?


Regards

Andreas Bluemle


-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

Re: LTTng tracing: hitting the message throttle

[Sage Weil]

Hi Andreas,

On Tue, 2 Dec 2014, Andreas Bluemle wrote:
> Hi,
> 
> while running tests to collect data using LTTng, I was hitting
> the message throttle in ceph-osd, which is controlled by configuration
> option osd_map_message_max, default 100.
> 
> If I am not mistaken, then this throttle hits when more than
> 100 messages received via SimpleMessenger are not yet destructed,
> i.e. are still under processing within the affected ceph-osd daemon.
> 
> I was running a fio test, random writes of 4 KByte with 16 parallel
> IO's. The storage cluster consists of 12 osds on 3 storage nodes,
> replication level 3.
> 
> I wonder why I hit that message throttle with my load profile:
> the 16 parallel IO's should at most generate 48 data messages and also
> 48 acknowledges across the cluster - so I would not expect anything
> even close to a 100 message limit on a single osd in my cluster.

I think the message may stay alive even after the client reply goes back 
because the op is written/committed to the journal but has not yet been 
written to the fs.  It may be that we can drop the reference at this 
point, although it may make the OpTracker output incomplete since the 
message is gone...

> Did anybody else experience such issues?
> 
> Are there other throttle values which I should look at?

There is a similar throttle for the number of bytes.

FWIW it may be that 100 is too low a default.  What value lets your 
workload run comfortably without throttling?

sage