Re: chaos monkeys

[Sam Lang <sam.lang@inktank.com>]

On 10/09/2012 12:16 PM, Gregory Farnum wrote:
> <also moved to ceph-devel>
> On Tue, Oct 9, 2012 at 9:59 AM, Sam Lang <sam.lang@inktank.com> wrote:
>> On 10/09/2012 11:46 AM, Gregory Farnum wrote:
>>>
>>> On Tue, Oct 9, 2012 at 9:43 AM, Sam Lang <sam.lang@inktank.com> wrote:
>>>>
>>>>
>>>> Could we add some other chaos monkeys to the network/storage
>>>> infrastructure
>>>> besides ms_inject_socket_failures?  In particular, I would like to add
>>>> ms_inject_delay_msg and ms_inject_reorder_msgs?  I think those could
>>>> potentially help flush out some bugs (such as:
>>>>
>>>> https://github.com/ceph/ceph/commit/fa66eaa162542ac01752ada91a46051dde060831).
>>>
>>>
>>> You're going to have to explain these more — ordered delivery over a
>>> connection is one of the guarantees that the messaging layer provides,
>>> so that doesn't sound like a configurable we're going to add.
>>
>>
>> That's true, but there's no guarantee that the source will always send them
>> in the same order.  The bug I linked above is a good example, the mds was
>> sending out two messages, one the open session reply, and another the stale
>> session async message.  The bug is only expressed when the stale comes
>> before the open session, which is possible in some cases.  The stale
>> originates from a timer expiring, and the open session is sent after the
>> journal commit, so the timing (and ordering) of those two messages can vary
>> based on when the timer thread gets scheduled to execute, how long the
>> journal commit takes, etc.
>>
>> Reordering messages at the destination would act to simulate all the
>> asynchronous paths like this that exist in our code.
>
> The sending messenger also maintains ordering invariants. The endpoint
> (the MDS) might not dispatch them in the same order all the time, but
> that's at a different semantic layer and is not something we can
> simulate inside the messenger — it requires semantic knowledge of
> which messages are okay to reorder. If we just did random reordering
> like you're suggesting, absolutely everything would break.

Putting a delay on the sender would avoid the reordering of messages 
that have semantic meaning but allow delay-caused reordering to occur 
for those that have no semantic dependency.

You're right that reordering at the receiver won't work, but it would be 
nice to have more concrete examples.  The only example I can come up 
with is the unsafe/safe messages from mds to client.  Even in that case 
it looks like we handle it by throwing away the unsafe message.  What 
other examples exist?  Caps issue/revoke?

-sam


>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html