set mds message priority to MSG_PRIO_HIGH

set mds message priority to MSG_PRIO_HIGH

[hjwsm1989-gmail]

Hi, 
I'm testing ceph with samba.
I have 20 OSD nodes on 4 hosts
dy01:1MON, 5 OSDs, 1 samba server
dy02: 1MDS, 5OSDs, 1 samba server
dy03: 5 OSDs, 1 samba server
dy04:5 OSDs, 1 samba server
every samba server exports the ceph kclient mount point.

When I set 3 samba clients connect to a samba server(dy01), and each write a
single 18GB file, gets 110MB/s speed each.
But when I use 4 clients, the cluster will report 'slow request' msg, and
this time I list files in samba client dir, it gets stuck.
And the test tool reports frame losing.
For the list operation related to MDS, I think the MDS op messages should
have higher priority than clients op.
Have I understood correctly? 

thanks

Re: set mds message priority to MSG_PRIO_HIGH

[Yan, Zheng]

On Sat, Oct 19, 2013 at 8:58 PM, hjwsm1989-gmail <hjwsm1989@gmail.com> wrote:
> Hi,
> I'm testing ceph with samba.
> I have 20 OSD nodes on 4 hosts
> dy01:1MON, 5 OSDs, 1 samba server
> dy02: 1MDS, 5OSDs, 1 samba server
> dy03: 5 OSDs, 1 samba server
> dy04:5 OSDs, 1 samba server
> every samba server exports the ceph kclient mount point.
>
> When I set 3 samba clients connect to a samba server(dy01), and each write a
> single 18GB file, gets 110MB/s speed each.
> But when I use 4 clients, the cluster will report 'slow request' msg, and
> this time I list files in samba client dir, it gets stuck.
> And the test tool reports frame losing.
> For the list operation related to MDS, I think the MDS op messages should
> have higher priority than clients op.
> Have I understood correctly?

Hi,

I think you are right, the MDS needs higher priority. could you modify
the mds code
and re-test.

In src/ceph_mds.cc, add
"messenger->set_default_send_priority(CEPH_MSG_PRIO_HIGH);"
after
"messenger->set_cluster_protocol(CEPH_MDS_PROTOCOL);"

Regards
Yan, Zheng
>
> thanks
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: set mds message priority to MSG_PRIO_HIGH

[Gregory Farnum]

On Sat, Oct 19, 2013 at 7:14 AM, Yan, Zheng <ukernel@gmail.com> wrote:
> On Sat, Oct 19, 2013 at 8:58 PM, hjwsm1989-gmail <hjwsm1989@gmail.com> wrote:
>> Hi,
>> I'm testing ceph with samba.
>> I have 20 OSD nodes on 4 hosts
>> dy01:1MON, 5 OSDs, 1 samba server
>> dy02: 1MDS, 5OSDs, 1 samba server
>> dy03: 5 OSDs, 1 samba server
>> dy04:5 OSDs, 1 samba server
>> every samba server exports the ceph kclient mount point.
>>
>> When I set 3 samba clients connect to a samba server(dy01), and each write a
>> single 18GB file, gets 110MB/s speed each.
>> But when I use 4 clients, the cluster will report 'slow request' msg, and
>> this time I list files in samba client dir, it gets stuck.
>> And the test tool reports frame losing.
>> For the list operation related to MDS, I think the MDS op messages should
>> have higher priority than clients op.
>> Have I understood correctly?
>
> Hi,
>
> I think you are right, the MDS needs higher priority. could you modify
> the mds code
> and re-test.
>
> In src/ceph_mds.cc, add
> "messenger->set_default_send_priority(CEPH_MSG_PRIO_HIGH);"
> after
> "messenger->set_cluster_protocol(CEPH_MDS_PROTOCOL);"
>
> Regards
> Yan, Zheng

If this actually turns out to be useful, we'll want to briefly go
through the code base and adjust the other levels — a quick grep
indicates the MDS sets some traffic to LOW that should correspondingly
be bumped up to NORMAL; the OSDs are currently using HIGH for acks and
commits, but we probably want those messages to continue having a
higher priority than any clients (MDS included) are setting messages
at.
(I'm not sure it actually will; I suspect what's happening here is
just that 4 clients are passing some limit that 3 aren't.)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html