* Client receives 'connection refused' only after heavy use
@ 2011-12-05 0:22 Noah Watkins
2011-12-05 1:03 ` Noah Watkins
0 siblings, 1 reply; 12+ messages in thread
From: Noah Watkins @ 2011-12-05 0:22 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]
We are experiencing client connection problems that occur only after
some period of heavy use. Prior to the 'connection refused' error in the
client log the cluster behaves as normal. Restarting Ceph solves the
problem but we are not able to finish long jobs.
Logs attached. I have the full 1 GB MDS log if needed, and included only
the portition of the log in which the client had problems plus about 5
seconds of context on either side of the test.
Thanks,
Noah
Client
====
...
2011-12-04 16:07:58.154523 7f4458314700 -- 192.168.141.123:0/1009375 >>
192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
l=0).connect 0
2011-12-04 16:07:58.154562 7f4458314700 -- 192.168.141.123:0/1009375 >>
192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:58.154605 7f4458314700 -- 192.168.141.123:0/1009375 >>
192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:58.154620 7f4458314700 -- 192.168.141.123:0/1009375 >>
192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
l=0).fault 111: Connection refused
2011-12-04 16:07:58.154635 7f4458314700 -- 192.168.141.123:0/1009375 >>
192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
l=0).fault waiting 3.200000
Full logs attached.
[-- Attachment #2: ceph.client.log --]
[-- Type: text/x-log, Size: 29471 bytes --]
2011-12-04 16:07:55.149410 7f44a328a700 -- :/0 register_entity client.?
2011-12-04 16:07:55.149459 7f44a328a700 -- :/0 register_entity client.? at :/0
2011-12-04 16:07:55.149554 7f44a328a700 -- :/0 messenger.start
2011-12-04 16:07:55.149632 7f44537fe700 -- :/1009375 reaper_entry start
2011-12-04 16:07:55.149686 7f44a328a700 -- :/1009375 ready :/1009375
2011-12-04 16:07:55.149717 7f44537fe700 -- :/1009375 reaper
2011-12-04 16:07:55.149736 7f44537fe700 -- :/1009375 reaper done
2011-12-04 16:07:55.149963 7f44a328a700 -- :/1009375 connect_rank to 192.168.141.123:6789/0, creating pipe and registering
2011-12-04 16:07:55.150048 7f44a328a700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=-1 pgs=0 cs=0 l=0).register_pipe
2011-12-04 16:07:55.150100 7f4458516700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:55.150134 7f44a328a700 -- :/1009375 --> 192.168.141.123:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x7f445437d020 con 0x7f445437ccc0
2011-12-04 16:07:55.150168 7f4458516700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=-1 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:55.150195 7f44a328a700 -- :/1009375 submit_message auth(proto 0 30 bytes) v1 remote 192.168.141.123:6789/0
2011-12-04 16:07:55.150225 7f4458516700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6789/0
2011-12-04 16:07:55.150530 7f4458516700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect read peer addr 192.168.141.123:6789/0 on socket 54
2011-12-04 16:07:55.150549 7f4458516700 -- :/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect peer addr for me is 192.168.141.123:36578/0
2011-12-04 16:07:55.150563 7f4458516700 -- 192.168.141.123:0/1009375 learned my addr 192.168.141.123:0/1009375
2011-12-04 16:07:55.150608 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect sent my addr 192.168.141.123:0/1009375
2011-12-04 16:07:55.150627 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect sending gseq=1 cseq=0 proto=15
2011-12-04 16:07:55.150661 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect wrote (self +) cseq, waiting for reply
2011-12-04 16:07:55.150697 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=0 cs=0 l=0).connect got reply tag 1 connect_seq 1 global_seq 1203 proto 15 flags 1
2011-12-04 16:07:55.150717 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).connect success 1, lossy = 1, features 4090
2011-12-04 16:07:55.150737 7f4458516700 -- 192.168.141.123:0/1009375 >> :/0 pipe(0x7f44542d4bf0 sd=-1 pgs=0 cs=0 l=0).queue_received queuing pipe
2011-12-04 16:07:55.150773 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).connect starting reader
2011-12-04 16:07:55.150820 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f44542d4bf0 dequeued 0
2011-12-04 16:07:55.150890 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.150924 7f4451ffb700 client.-1 ms_handle_connect on 192.168.141.123:6789/0
2011-12-04 16:07:55.150953 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer encoding 1 0x7f445437d020 auth(proto 0 30 bytes) v1
2011-12-04 16:07:55.150991 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.151022 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sending 1 0x7f445437d020
2011-12-04 16:07:55.151042 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_message 0x7f445437d020
2011-12-04 16:07:55.151094 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.151109 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.151334 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got ACK
2011-12-04 16:07:55.151367 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.151388 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got MSG
2011-12-04 16:07:55.151423 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got envelope type=18 src mon.0 front=24 data=0 off 0
2011-12-04 16:07:55.151439 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader wants 24 from dispatch throttler 0/104857600
2011-12-04 16:07:55.151467 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got front 24
2011-12-04 16:07:55.151489 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).aborted = 0
2011-12-04 16:07:55.151504 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got 24 + 0 + 0 byte message
2011-12-04 16:07:55.151590 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got message 1 0x422fbc50 auth_reply(proto 1 0 Success) v1
2011-12-04 16:07:55.151618 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).queue_received queuing pipe
2011-12-04 16:07:55.151653 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.151684 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.151710 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_ack 1
2011-12-04 16:07:55.151732 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f445437ca50 dequeued 0x422fbc50
2011-12-04 16:07:55.151758 7f4451ffb700 -- 192.168.141.123:0/1009375 <== mon.0 192.168.141.123:6789/0 1 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (1949490694 0 0) 0x422fbc50 con 0x7f445437ccc0
2011-12-04 16:07:55.151782 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.151800 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.151846 7f4451ffb700 -- 192.168.141.123:0/1009375 --> 192.168.141.123:6789/0 -- mon_subscribe({mdsmap=0+,monmap=0+,osdmap=0}) v1 -- ?+0 0x7f445437d230 con 0x7f445437ccc0
2011-12-04 16:07:55.151865 7f4451ffb700 -- 192.168.141.123:0/1009375 submit_message mon_subscribe({mdsmap=0+,monmap=0+,osdmap=0}) v1 remote 192.168.141.123:6789/0
2011-12-04 16:07:55.151889 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.151916 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer encoding 2 0x7f445437d230 mon_subscribe({mdsmap=0+,monmap=0+,osdmap=0}) v1
2011-12-04 16:07:55.151948 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_throttle_release 24 to dispatch throttler 24/104857600
2011-12-04 16:07:55.151971 7f4451ffb700 -- 192.168.141.123:0/1009375 done calling dispatch on 0x422fbc50
2011-12-04 16:07:55.151997 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sending 2 0x7f445437d230
2011-12-04 16:07:55.152018 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_message 0x7f445437d230
2011-12-04 16:07:55.152041 7f44a328a700 client.6697 mounted: have osdmap 0 and mdsmap 0
2011-12-04 16:07:55.152088 7f44a328a700 client.6697 mds is 0
2011-12-04 16:07:55.152098 7f44a328a700 client.6697 target mds.0 not active, waiting for new mdsmap
2011-12-04 16:07:55.152151 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.152168 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.152262 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got ACK
2011-12-04 16:07:55.152288 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.152306 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got MSG
2011-12-04 16:07:55.152327 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got envelope type=21 src mon.0 front=535 data=0 off 0
2011-12-04 16:07:55.152341 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader wants 535 from dispatch throttler 0/104857600
2011-12-04 16:07:55.152363 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got front 535
2011-12-04 16:07:55.152383 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).aborted = 0
2011-12-04 16:07:55.152396 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got 535 + 0 + 0 byte message
2011-12-04 16:07:55.152432 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got message 2 0x422fbc50 mdsmap(e 14) v1
2011-12-04 16:07:55.152453 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).queue_received queuing pipe
2011-12-04 16:07:55.152483 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.152509 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.152533 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_ack 2
2011-12-04 16:07:55.152551 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f445437ca50 dequeued 0x422fbc50
2011-12-04 16:07:55.152579 7f4451ffb700 -- 192.168.141.123:0/1009375 <== mon.0 192.168.141.123:6789/0 2 ==== mdsmap(e 14) v1 ==== 535+0+0 (3446747908 0 0) 0x422fbc50 con 0x7f445437ccc0
2011-12-04 16:07:55.152605 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.152626 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.152648 7f4451ffb700 client.6697 handle_mds_map epoch 14
2011-12-04 16:07:55.152678 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got MSG
2011-12-04 16:07:55.152726 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got envelope type=4 src mon.0 front=187 data=0 off 0
2011-12-04 16:07:55.152742 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader wants 187 from dispatch throttler 535/104857600
2011-12-04 16:07:55.152757 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_throttle_release 535 to dispatch throttler 722/104857600
2011-12-04 16:07:55.152776 7f4451ffb700 -- 192.168.141.123:0/1009375 done calling dispatch on 0x422fbc50
2011-12-04 16:07:55.152796 7f44a328a700 client.6697 mds is 0
2011-12-04 16:07:55.152817 7f44a328a700 client.6697 opening session to mds.0
2011-12-04 16:07:55.152843 7f44a328a700 -- 192.168.141.123:0/1009375 --> mds.0 192.168.141.123:6800/1325 -- client_session(request_open) v1 -- ?+0 0x7f44544cd800
2011-12-04 16:07:55.152867 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got front 187
2011-12-04 16:07:55.152897 7f44a328a700 -- 192.168.141.123:0/1009375 submit_message client_session(request_open) v1 remote, 192.168.141.123:6800/1325, new pipe.
2011-12-04 16:07:55.152916 7f44a328a700 -- 192.168.141.123:0/1009375 connect_rank to 192.168.141.123:6800/1325, creating pipe and registering
2011-12-04 16:07:55.152940 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).aborted = 0
2011-12-04 16:07:55.152960 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got 187 + 0 + 0 byte message
2011-12-04 16:07:55.152979 7f44a328a700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=-1 pgs=0 cs=0 l=0).register_pipe
2011-12-04 16:07:55.153018 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=-1 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:55.153051 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=-1 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:55.153079 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got message 3 0x422fbc50 mon_map v1
2011-12-04 16:07:55.153109 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).queue_received queuing pipe
2011-12-04 16:07:55.153128 7f44a328a700 client.6697 waiting for session to mds.0 to open
2011-12-04 16:07:55.153162 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:55.153189 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.153225 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:55.153253 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f445437ca50 dequeued 0x422fbc50
2011-12-04 16:07:55.153281 7f4451ffb700 -- 192.168.141.123:0/1009375 <== mon.0 192.168.141.123:6789/0 3 ==== mon_map v1 ==== 187+0+0 (1658340354 0 0) 0x422fbc50 con 0x7f445437ccc0
2011-12-04 16:07:55.153307 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:55.153324 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_throttle_release 187 to dispatch throttler 187/104857600
2011-12-04 16:07:55.153342 7f4451ffb700 -- 192.168.141.123:0/1009375 done calling dispatch on 0x422fbc50
2011-12-04 16:07:55.153366 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got MSG
2011-12-04 16:07:55.153406 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got envelope type=41 src mon.0 front=6900 data=0 off 0
2011-12-04 16:07:55.153425 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader wants 6900 from dispatch throttler 0/104857600
2011-12-04 16:07:55.153452 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault first fault
2011-12-04 16:07:55.153474 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:55.153491 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:55.153515 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.153536 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_ack 3
2011-12-04 16:07:55.153558 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.153575 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.153599 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:55.153627 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got front 6900
2011-12-04 16:07:55.153647 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:55.153669 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:55.153695 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault waiting 0.200000
2011-12-04 16:07:55.153721 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).aborted = 0
2011-12-04 16:07:55.153738 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got 6900 + 0 + 0 byte message
2011-12-04 16:07:55.153794 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got message 4 0x4227d030 osd_map(48..48 src has 1..48) v2
2011-12-04 16:07:55.153815 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).queue_received queuing pipe
2011-12-04 16:07:55.153838 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.153864 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f445437ca50 dequeued 0x4227d030
2011-12-04 16:07:55.153891 7f4451ffb700 -- 192.168.141.123:0/1009375 <== mon.0 192.168.141.123:6789/0 4 ==== osd_map(48..48 src has 1..48) v2 ==== 6900+0+0 (1758851646 0 0) 0x4227d030 con 0x7f445437ccc0
2011-12-04 16:07:55.153915 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got MSG
2011-12-04 16:07:55.153945 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.153969 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_ack 4
2011-12-04 16:07:55.154011 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got envelope type=16 src mon.0 front=20 data=0 off 0
2011-12-04 16:07:55.154033 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader wants 20 from dispatch throttler 6900/104857600
2011-12-04 16:07:55.154057 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.154078 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.154102 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got front 20
2011-12-04 16:07:55.154126 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).aborted = 0
2011-12-04 16:07:55.154144 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got 20 + 0 + 0 byte message
2011-12-04 16:07:55.154167 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_throttle_release 6900 to dispatch throttler 6920/104857600
2011-12-04 16:07:55.154187 7f4451ffb700 -- 192.168.141.123:0/1009375 done calling dispatch on 0x4227d030
2011-12-04 16:07:55.154212 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader got message 5 0x7f44543e82c0 mon_subscribe_ack(300s) v1
2011-12-04 16:07:55.154236 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).queue_received queuing pipe
2011-12-04 16:07:55.154260 7f4458415700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).reader reading tag...
2011-12-04 16:07:55.154283 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_entry pipe 0x7f445437ca50 dequeued 0x7f44543e82c0
2011-12-04 16:07:55.154306 7f4451ffb700 -- 192.168.141.123:0/1009375 <== mon.0 192.168.141.123:6789/0 5 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1384836696 0 0) 0x7f44543e82c0 con 0x7f445437ccc0
2011-12-04 16:07:55.154329 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.154349 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_ack 5
2011-12-04 16:07:55.154368 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:55.154385 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:55.154406 7f4451ffb700 -- 192.168.141.123:0/1009375 dispatch_throttle_release 20 to dispatch throttler 20/104857600
2011-12-04 16:07:55.154419 7f4451ffb700 -- 192.168.141.123:0/1009375 done calling dispatch on 0x7f44543e82c0
2011-12-04 16:07:55.353786 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault done waiting or woke up
2011-12-04 16:07:55.353808 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:55.353821 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:55.353847 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:55.353891 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:55.353916 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:55.353931 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault waiting 0.400000
2011-12-04 16:07:55.754011 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault done waiting or woke up
2011-12-04 16:07:55.754032 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:55.754045 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:55.754070 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:55.754109 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:55.754125 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:55.754139 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault waiting 0.800000
2011-12-04 16:07:56.152068 7f4452ffd700 client.6697 renew_caps()
2011-12-04 16:07:56.554233 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault done waiting or woke up
2011-12-04 16:07:56.554268 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:56.554281 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:56.554320 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:56.554381 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:56.554398 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:56.554413 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault waiting 1.600000
2011-12-04 16:07:58.149960 7f44517fa700 -- 192.168.141.123:0/1009375 send_keepalive con 0x7f445437ccc0, have pipe.
2011-12-04 16:07:58.150013 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:58.150051 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).write_keepalive
2011-12-04 16:07:58.150098 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer: state = 2 policy.server=0
2011-12-04 16:07:58.150113 7f4458516700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6789/0 pipe(0x7f445437ca50 sd=54 pgs=1203 cs=1 l=1).writer sleeping
2011-12-04 16:07:58.154492 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault done waiting or woke up
2011-12-04 16:07:58.154510 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).writer: state = 1 policy.server=0
2011-12-04 16:07:58.154523 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect 0
2011-12-04 16:07:58.154562 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connecting to 192.168.141.123:6800/1325
2011-12-04 16:07:58.154605 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
2011-12-04 16:07:58.154620 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault 111: Connection refused
2011-12-04 16:07:58.154635 7f4458314700 -- 192.168.141.123:0/1009375 >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault waiting 3.200000
[-- Attachment #3: mds.conn_ref.log --]
[-- Type: text/x-log, Size: 6915 bytes --]
2011-12-04 16:13:15.385981 7f1403bc5700 mds.0.locker scatter_tick
2011-12-04 16:13:15.385996 7f1403bc5700 mds.0.server find_idle_sessions. laggy until 0.000000
2011-12-04 16:13:15.386014 7f1403bc5700 mds.0.bal tick last_sample now 2011-12-04 16:13:15.386009
2011-12-04 16:13:15.386058 7f1403bc5700 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.01>
2011-12-04 16:13:15.386095 7f1403bc5700 mds.0.bal mds.0 epoch 1645 load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.01>
2011-12-04 16:13:15.386109 7f1403bc5700 mds.0.snap check_osd_map - version unchanged
2011-12-04 16:13:16.645132 7f1403bc5700 mds.0.3 beacon_send up:active seq 4130 (currently up:active)
2011-12-04 16:13:16.645175 7f1403bc5700 -- 192.168.141.123:6800/1325 --> 192.168.141.123:6789/0 -- mdsbeacon(5497/a up:active seq 4130 v14) v1 -- ?+0 0x28ff500 con 0x1b518c0
2011-12-04 16:13:16.645529 7f1405ac9700 -- 192.168.141.123:6800/1325 <== mon.0 192.168.141.123:6789/0 4260 ==== mdsbeacon(5497/a up:active seq 4130 v14) v2 ==== 103+0+0 (1047759573 0 0) 0x20f4500 con 0x1b518c0
2011-12-04 16:13:16.645567 7f1405ac9700 mds.0.3 handle_mds_beacon up:active seq 4130 rtt 0.000413
2011-12-04 16:13:20.376569 7f1403bc5700 mds.0.cache trim max=100000 cur=2220
2011-12-04 16:13:20.376602 7f1403bc5700 mds.0.cache trim_client_leases
2011-12-04 16:13:20.385490 7f1403bc5700 mds.0.cache check_memory_usage total 2247420, rss 60632, heap -2019376, malloc 15008 mmap 0, baseline 115192, buffers 0, max 1048576, 0 / 488 inodes have caps, 0 caps, 0 caps per inode
2011-12-04 16:13:20.386041 7f1403bc5700 mds.0.log trim 8 / 30 segments, 12919 / -1 events, 0 (0) expiring, 0 (0) expired
2011-12-04 16:13:20.386080 7f1403bc5700 mds.0.log _trim_expired_segments waiting for 4194304 to expire
2011-12-04 16:13:20.386190 7f1403bc5700 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.01>
2011-12-04 16:13:20.386222 7f1403bc5700 mds.0.locker scatter_tick
2011-12-04 16:13:20.386244 7f1403bc5700 mds.0.server find_idle_sessions. laggy until 0.000000
2011-12-04 16:13:20.386265 7f1403bc5700 mds.0.bal tick last_sample now 2011-12-04 16:13:20.386259
2011-12-04 16:13:20.386280 7f1403bc5700 mds.0.snap check_osd_map - version unchanged
2011-12-04 16:13:20.645259 7f1403bc5700 mds.0.3 beacon_send up:active seq 4131 (currently up:active)
2011-12-04 16:13:20.645307 7f1403bc5700 -- 192.168.141.123:6800/1325 --> 192.168.141.123:6789/0 -- mdsbeacon(5497/a up:active seq 4131 v14) v1 -- ?+0 0x279a780 con 0x1b518c0
2011-12-04 16:13:20.645617 7f1405ac9700 -- 192.168.141.123:6800/1325 <== mon.0 192.168.141.123:6789/0 4261 ==== mdsbeacon(5497/a up:active seq 4131 v14) v2 ==== 103+0+0 (1197143986 0 0) 0x20f4780 con 0x1b518c0
2011-12-04 16:13:20.645646 7f1405ac9700 mds.0.3 handle_mds_beacon up:active seq 4131 rtt 0.000363
2011-12-04 16:13:24.645420 7f1403bc5700 mds.0.3 beacon_send up:active seq 4132 (currently up:active)
2011-12-04 16:13:24.645485 7f1403bc5700 -- 192.168.141.123:6800/1325 --> 192.168.141.123:6789/0 -- mdsbeacon(5497/a up:active seq 4132 v14) v1 -- ?+0 0x279aa00 con 0x1b518c0
2011-12-04 16:13:24.645900 7f1405ac9700 -- 192.168.141.123:6800/1325 <== mon.0 192.168.141.123:6789/0 4262 ==== mdsbeacon(5497/a up:active seq 4132 v14) v2 ==== 103+0+0 (761852790 0 0) 0x2928280 con 0x1b518c0
2011-12-04 16:13:24.645960 7f1405ac9700 mds.0.3 handle_mds_beacon up:active seq 4132 rtt 0.000496
2011-12-04 16:13:25.376640 7f1403bc5700 mds.0.cache trim max=100000 cur=2220
2011-12-04 16:13:25.376668 7f1403bc5700 mds.0.cache trim_client_leases
2011-12-04 16:13:25.385983 7f1403bc5700 mds.0.cache check_memory_usage total 2247420, rss 60632, heap -2019376, malloc 15008 mmap 0, baseline 115192, buffers 0, max 1048576, 0 / 488 inodes have caps, 0 caps, 0 caps per inode
2011-12-04 16:13:25.386029 7f1403bc5700 mds.0.log trim 8 / 30 segments, 12919 / -1 events, 0 (0) expiring, 0 (0) expired
2011-12-04 16:13:25.386047 7f1403bc5700 mds.0.log _trim_expired_segments waiting for 4194304 to expire
2011-12-04 16:13:25.386124 7f1403bc5700 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.25>
2011-12-04 16:13:25.386156 7f1403bc5700 mds.0.locker scatter_tick
2011-12-04 16:13:25.386180 7f1403bc5700 mds.0.server find_idle_sessions. laggy until 0.000000
2011-12-04 16:13:25.386205 7f1403bc5700 mds.0.bal tick last_sample now 2011-12-04 16:13:25.386199
2011-12-04 16:13:25.386257 7f1403bc5700 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.25>
2011-12-04 16:13:25.386300 7f1403bc5700 mds.0.bal mds.0 epoch 1646 load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.25>
2011-12-04 16:13:25.386321 7f1403bc5700 mds.0.snap check_osd_map - version unchanged
2011-12-04 16:13:28.645586 7f1403bc5700 mds.0.3 beacon_send up:active seq 4133 (currently up:active)
2011-12-04 16:13:28.645642 7f1403bc5700 -- 192.168.141.123:6800/1325 --> 192.168.141.123:6789/0 -- mdsbeacon(5497/a up:active seq 4133 v14) v1 -- ?+0 0x279ac80 con 0x1b518c0
2011-12-04 16:13:28.646032 7f1405ac9700 -- 192.168.141.123:6800/1325 <== mon.0 192.168.141.123:6789/0 4263 ==== mdsbeacon(5497/a up:active seq 4133 v14) v2 ==== 103+0+0 (1413582353 0 0) 0x2928a00 con 0x1b518c0
2011-12-04 16:13:28.646077 7f1405ac9700 mds.0.3 handle_mds_beacon up:active seq 4133 rtt 0.000458
2011-12-04 16:13:30.376715 7f1403bc5700 mds.0.cache trim max=100000 cur=2220
2011-12-04 16:13:30.376743 7f1403bc5700 mds.0.cache trim_client_leases
2011-12-04 16:13:30.385607 7f1403bc5700 mds.0.cache check_memory_usage total 2247420, rss 60632, heap -2019376, malloc 15008 mmap 0, baseline 115192, buffers 0, max 1048576, 0 / 488 inodes have caps, 0 caps, 0 caps per inode
2011-12-04 16:13:30.386317 7f1403bc5700 mds.0.log trim 8 / 30 segments, 12919 / -1 events, 0 (0) expiring, 0 (0) expired
2011-12-04 16:13:30.386358 7f1403bc5700 mds.0.log _trim_expired_segments waiting for 4194304 to expire
2011-12-04 16:13:30.386432 7f1403bc5700 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0.23>
2011-12-04 16:13:30.386463 7f1403bc5700 mds.0.locker scatter_tick
2011-12-04 16:13:30.386485 7f1403bc5700 mds.0.server find_idle_sessions. laggy until 0.000000
2011-12-04 16:13:30.386505 7f1403bc5700 mds.0.bal tick last_sample now 2011-12-04 16:13:30.386500
2011-12-04 16:13:30.386521 7f1403bc5700 mds.0.snap check_osd_map - version unchanged
2011-12-04 16:13:32.645749 7f1403bc5700 mds.0.3 beacon_send up:active seq 4134 (currently up:active)
2011-12-04 16:13:32.645810 7f1403bc5700 -- 192.168.141.123:6800/1325 --> 192.168.141.123:6789/0 -- mdsbeacon(5497/a up:active seq 4134 v14) v1 -- ?+0 0x2826780 con 0x1b518c0
2011-12-04 16:13:32.646164 7f1405ac9700 -- 192.168.141.123:6800/1325 <== mon.0 192.168.141.123:6789/0 4264 ==== mdsbeacon(5497/a up:active seq 4134 v14) v2 ==== 103+0+0 (3745122744 0 0) 0x2928c80 con 0x1b518c0
2011-12-04 16:13:32.646221 7f1405ac9700 mds.0.3 handle_mds_beacon up:active seq 4134 rtt 0.000434
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 0:22 Client receives 'connection refused' only after heavy use Noah Watkins
@ 2011-12-05 1:03 ` Noah Watkins
2011-12-05 4:47 ` Sage Weil
0 siblings, 1 reply; 12+ messages in thread
From: Noah Watkins @ 2011-12-05 1:03 UTC (permalink / raw)
To: ceph-devel
Yikes, I think this was actually the problem. nm
# ulimit -n
1024
-----
root@issdm-23:/var/log/ceph# grep -n "Too many" full_conn_refused.log
2417924:2011-12-04 14:52:15.289873 7f1406ecb700 --
192.168.141.123:6800/1325 accepter no incoming connection? sd = -1
errno 24 Too many open files
2417925:2011-12-04 14:52:15.289923 7f1406ecb700 --
192.168.141.123:6800/1325 accepter no incoming connection? sd = -1
errno 24 Too many open files
2417926:2011-12-04 14:52:15.289952 7f1406ecb700 --
192.168.141.123:6800/1325 accepter no incoming connection? sd = -1
errno 24 Too many open files
2417927:2011-12-04 14:52:15.289970 7f1406ecb700 --
192.168.141.123:6800/1325 accepter no incoming connection? sd = -1
errno 24 Too many open files
2417928:2011-12-04 14:52:15.290002 7f1406ecb700 --
192.168.141.123:6800/1325 accepter no incoming connection? sd = -1
errno 24 Too many open files
On 12/04/2011 04:22 PM, Noah Watkins wrote:
> We are experiencing client connection problems that occur only after
> some period of heavy use. Prior to the 'connection refused' error in
> the client log the cluster behaves as normal. Restarting Ceph solves
> the problem but we are not able to finish long jobs.
>
> Logs attached. I have the full 1 GB MDS log if needed, and included
> only the portition of the log in which the client had problems plus
> about 5 seconds of context on either side of the test.
>
> Thanks,
> Noah
>
> Client
> ====
> ...
> 2011-12-04 16:07:58.154523 7f4458314700 -- 192.168.141.123:0/1009375
> >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> l=0).connect 0
> 2011-12-04 16:07:58.154562 7f4458314700 -- 192.168.141.123:0/1009375
> >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> l=0).connecting to 192.168.141.123:6800/1325
> 2011-12-04 16:07:58.154605 7f4458314700 -- 192.168.141.123:0/1009375
> >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> l=0).connect error 192.168.141.123:6800/1325, 111: Connection refused
> 2011-12-04 16:07:58.154620 7f4458314700 -- 192.168.141.123:0/1009375
> >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> l=0).fault 111: Connection refused
> 2011-12-04 16:07:58.154635 7f4458314700 -- 192.168.141.123:0/1009375
> >> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> l=0).fault waiting 3.200000
>
> Full logs attached.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 1:03 ` Noah Watkins
@ 2011-12-05 4:47 ` Sage Weil
2011-12-05 6:22 ` Noah Watkins
2011-12-05 18:57 ` Noah Watkins
0 siblings, 2 replies; 12+ messages in thread
From: Sage Weil @ 2011-12-05 4:47 UTC (permalink / raw)
To: Noah Watkins; +Cc: ceph-devel
On Sun, 4 Dec 2011, Noah Watkins wrote:
> Yikes, I think this was actually the problem. nm
>
> # ulimit -n
> 1024
I'm a little surprised the fd count got that high with a fixed size
cluster. Were there lots of short-lived clients?
It would be interested to see what `ls -al /proc/$pid/fd` looks like after
the process has been running for a while... there is probably a leak
somewhere.
sage
>
> -----
>
> root@issdm-23:/var/log/ceph# grep -n "Too many" full_conn_refused.log
> 2417924:2011-12-04 14:52:15.289873 7f1406ecb700 -- 192.168.141.123:6800/1325
> accepter no incoming connection? sd = -1 errno 24 Too many open files
> 2417925:2011-12-04 14:52:15.289923 7f1406ecb700 -- 192.168.141.123:6800/1325
> accepter no incoming connection? sd = -1 errno 24 Too many open files
> 2417926:2011-12-04 14:52:15.289952 7f1406ecb700 -- 192.168.141.123:6800/1325
> accepter no incoming connection? sd = -1 errno 24 Too many open files
> 2417927:2011-12-04 14:52:15.289970 7f1406ecb700 -- 192.168.141.123:6800/1325
> accepter no incoming connection? sd = -1 errno 24 Too many open files
> 2417928:2011-12-04 14:52:15.290002 7f1406ecb700 -- 192.168.141.123:6800/1325
> accepter no incoming connection? sd = -1 errno 24 Too many open files
>
> On 12/04/2011 04:22 PM, Noah Watkins wrote:
> > We are experiencing client connection problems that occur only after some
> > period of heavy use. Prior to the 'connection refused' error in the client
> > log the cluster behaves as normal. Restarting Ceph solves the problem but we
> > are not able to finish long jobs.
> >
> > Logs attached. I have the full 1 GB MDS log if needed, and included only the
> > portition of the log in which the client had problems plus about 5 seconds
> > of context on either side of the test.
> >
> > Thanks,
> > Noah
> >
> > Client
> > ====
> > ...
> > 2011-12-04 16:07:58.154523 7f4458314700 -- 192.168.141.123:0/1009375 >>
> > 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect
> > 0
> > 2011-12-04 16:07:58.154562 7f4458314700 -- 192.168.141.123:0/1009375 >>
> > 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
> > l=0).connecting to 192.168.141.123:6800/1325
> > 2011-12-04 16:07:58.154605 7f4458314700 -- 192.168.141.123:0/1009375 >>
> > 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect
> > error 192.168.141.123:6800/1325, 111: Connection refused
> > 2011-12-04 16:07:58.154620 7f4458314700 -- 192.168.141.123:0/1009375 >>
> > 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault
> > 111: Connection refused
> > 2011-12-04 16:07:58.154635 7f4458314700 -- 192.168.141.123:0/1009375 >>
> > 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault
> > waiting 3.200000
> >
> > Full logs attached.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 4:47 ` Sage Weil
@ 2011-12-05 6:22 ` Noah Watkins
2011-12-05 18:57 ` Noah Watkins
1 sibling, 0 replies; 12+ messages in thread
From: Noah Watkins @ 2011-12-05 6:22 UTC (permalink / raw)
To: Sage Weil; +Cc: Noah Watkins, ceph-devel@vger.kernel.org
Sent from my iPhone
On Dec 4, 2011, at 20:48, Sage Weil <sage@newdream.net> wrote:
> On Sun, 4 Dec 2011, Noah Watkins wrote:
>> Yikes, I think this was actually the problem. nm
>>
>> # ulimit -n
>> 1024
>
> I'm a little surprised the fd count got that high with a fixed size
> cluster. Were there lots of short-lived clients?
Not a lot. Maybe a hundred total over a few hours.
>
> It would be interested to see what `ls -al /proc/$pid/fd` looks like after
> the process has been running for a while... there is probably a leak
> somewhere.
I checked this out after the problem became noticeable. There were
significantly less than 1024 file nos but still several hundred w no
active clients. I think this latest fix is masking things. I'll drop
the ulimit backdown and gather some more info.
>
> sage
>
>
>>
>> -----
>>
>> root@issdm-23:/var/log/ceph# grep -n "Too many" full_conn_refused.log
>> 2417924:2011-12-04 14:52:15.289873 7f1406ecb700 -- 192.168.141.123:6800/1325
>> accepter no incoming connection? sd = -1 errno 24 Too many open files
>> 2417925:2011-12-04 14:52:15.289923 7f1406ecb700 -- 192.168.141.123:6800/1325
>> accepter no incoming connection? sd = -1 errno 24 Too many open files
>> 2417926:2011-12-04 14:52:15.289952 7f1406ecb700 -- 192.168.141.123:6800/1325
>> accepter no incoming connection? sd = -1 errno 24 Too many open files
>> 2417927:2011-12-04 14:52:15.289970 7f1406ecb700 -- 192.168.141.123:6800/1325
>> accepter no incoming connection? sd = -1 errno 24 Too many open files
>> 2417928:2011-12-04 14:52:15.290002 7f1406ecb700 -- 192.168.141.123:6800/1325
>> accepter no incoming connection? sd = -1 errno 24 Too many open files
>>
>> On 12/04/2011 04:22 PM, Noah Watkins wrote:
>>> We are experiencing client connection problems that occur only after some
>>> period of heavy use. Prior to the 'connection refused' error in the client
>>> log the cluster behaves as normal. Restarting Ceph solves the problem but we
>>> are not able to finish long jobs.
>>>
>>> Logs attached. I have the full 1 GB MDS log if needed, and included only the
>>> portition of the log in which the client had problems plus about 5 seconds
>>> of context on either side of the test.
>>>
>>> Thanks,
>>> Noah
>>>
>>> Client
>>> ====
>>> ...
>>> 2011-12-04 16:07:58.154523 7f4458314700 -- 192.168.141.123:0/1009375 >>
>>> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect
>>> 0
>>> 2011-12-04 16:07:58.154562 7f4458314700 -- 192.168.141.123:0/1009375 >>
>>> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0
>>> l=0).connecting to 192.168.141.123:6800/1325
>>> 2011-12-04 16:07:58.154605 7f4458314700 -- 192.168.141.123:0/1009375 >>
>>> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).connect
>>> error 192.168.141.123:6800/1325, 111: Connection refused
>>> 2011-12-04 16:07:58.154620 7f4458314700 -- 192.168.141.123:0/1009375 >>
>>> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault
>>> 111: Connection refused
>>> 2011-12-04 16:07:58.154635 7f4458314700 -- 192.168.141.123:0/1009375 >>
>>> 192.168.141.123:6800/1325 pipe(0x7f445437d020 sd=55 pgs=0 cs=0 l=0).fault
>>> waiting 3.200000
>>>
>>> Full logs attached.
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 4:47 ` Sage Weil
2011-12-05 6:22 ` Noah Watkins
@ 2011-12-05 18:57 ` Noah Watkins
2011-12-05 19:03 ` Tommi Virtanen
1 sibling, 1 reply; 12+ messages in thread
From: Noah Watkins @ 2011-12-05 18:57 UTC (permalink / raw)
To: Sage Weil; +Cc: Noah Watkins, ceph-devel
On 12/04/2011 08:47 PM, Sage Weil wrote:
> It would be interested to see what `ls -al /proc/$pid/fd` looks like after
> the process has been running for a while... there is probably a leak
> somewhere.
>
Here is the proc/<mds_pid>/fd contents. These contents should reflect an
idle cluster with no active clients.
-Noah
root@issdm-23:~# ls -la /proc/1394/fd | wc -l
542
root@issdm-23:~# ls -la /proc/1394/fd/
total 0
dr-x------ 2 root root 0 2011-12-05 10:54 .
dr-xr-xr-x 7 root root 0 2011-12-04 16:56 ..
lr-x------ 1 root root 64 2011-12-05 10:54 0 -> /dev/null
lr-x------ 1 root root 64 2011-12-05 10:54 1 -> /dev/null
lrwx------ 1 root root 64 2011-12-05 10:54 10 -> socket:[40002]
lrwx------ 1 root root 64 2011-12-05 10:54 100 -> socket:[12769]
lrwx------ 1 root root 64 2011-12-05 10:54 101 -> socket:[12770]
lrwx------ 1 root root 64 2011-12-05 10:54 102 -> socket:[12773]
lrwx------ 1 root root 64 2011-12-05 10:54 103 -> socket:[12774]
lrwx------ 1 root root 64 2011-12-05 10:54 104 -> socket:[12776]
lrwx------ 1 root root 64 2011-12-05 10:54 105 -> socket:[12786]
lrwx------ 1 root root 64 2011-12-05 10:54 106 -> socket:[12787]
lrwx------ 1 root root 64 2011-12-05 10:54 107 -> socket:[12788]
lrwx------ 1 root root 64 2011-12-05 10:54 108 -> socket:[12789]
lrwx------ 1 root root 64 2011-12-05 10:54 109 -> socket:[12790]
lrwx------ 1 root root 64 2011-12-05 10:54 11 -> socket:[40004]
lrwx------ 1 root root 64 2011-12-05 10:54 110 -> socket:[12803]
lrwx------ 1 root root 64 2011-12-05 10:54 111 -> socket:[12804]
lrwx------ 1 root root 64 2011-12-05 10:54 112 -> socket:[12816]
lrwx------ 1 root root 64 2011-12-05 10:54 113 -> socket:[12818]
lrwx------ 1 root root 64 2011-12-05 10:54 114 -> socket:[12831]
lrwx------ 1 root root 64 2011-12-05 10:54 115 -> socket:[12832]
lrwx------ 1 root root 64 2011-12-05 10:54 116 -> socket:[12834]
lrwx------ 1 root root 64 2011-12-05 10:54 117 -> socket:[12836]
lrwx------ 1 root root 64 2011-12-05 10:54 118 -> socket:[12838]
lrwx------ 1 root root 64 2011-12-05 10:54 119 -> socket:[12840]
lrwx------ 1 root root 64 2011-12-05 10:54 12 -> socket:[11560]
lrwx------ 1 root root 64 2011-12-05 10:54 120 -> socket:[12842]
lrwx------ 1 root root 64 2011-12-05 10:54 121 -> socket:[12844]
lrwx------ 1 root root 64 2011-12-05 10:54 122 -> socket:[12846]
lrwx------ 1 root root 64 2011-12-05 10:54 123 -> socket:[12848]
lrwx------ 1 root root 64 2011-12-05 10:54 124 -> socket:[12851]
lrwx------ 1 root root 64 2011-12-05 10:54 125 -> socket:[12852]
lrwx------ 1 root root 64 2011-12-05 10:54 126 -> socket:[12854]
lrwx------ 1 root root 64 2011-12-05 10:54 127 -> socket:[12856]
lrwx------ 1 root root 64 2011-12-05 10:54 128 -> socket:[12858]
lrwx------ 1 root root 64 2011-12-05 10:54 129 -> socket:[12860]
lrwx------ 1 root root 64 2011-12-05 10:54 13 -> socket:[39999]
lrwx------ 1 root root 64 2011-12-05 10:54 130 -> socket:[12862]
lrwx------ 1 root root 64 2011-12-05 10:54 131 -> socket:[12864]
lrwx------ 1 root root 64 2011-12-05 10:54 132 -> socket:[12866]
lrwx------ 1 root root 64 2011-12-05 10:54 133 -> socket:[12873]
lrwx------ 1 root root 64 2011-12-05 10:54 134 -> socket:[12880]
lrwx------ 1 root root 64 2011-12-05 10:54 135 -> socket:[12882]
lrwx------ 1 root root 64 2011-12-05 10:54 136 -> socket:[12894]
lrwx------ 1 root root 64 2011-12-05 10:54 137 -> socket:[12896]
lrwx------ 1 root root 64 2011-12-05 10:54 138 -> socket:[12908]
lrwx------ 1 root root 64 2011-12-05 10:54 139 -> socket:[12910]
lrwx------ 1 root root 64 2011-12-05 10:54 14 -> socket:[40005]
lrwx------ 1 root root 64 2011-12-05 10:54 140 -> socket:[14170]
lrwx------ 1 root root 64 2011-12-05 10:54 141 -> socket:[12915]
lrwx------ 1 root root 64 2011-12-05 10:54 142 -> socket:[12916]
lrwx------ 1 root root 64 2011-12-05 10:54 143 -> socket:[12918]
lrwx------ 1 root root 64 2011-12-05 10:54 144 -> socket:[12920]
lrwx------ 1 root root 64 2011-12-05 10:54 145 -> socket:[12927]
lrwx------ 1 root root 64 2011-12-05 10:54 146 -> socket:[12944]
lrwx------ 1 root root 64 2011-12-05 10:54 147 -> socket:[12956]
lrwx------ 1 root root 64 2011-12-05 10:54 148 -> socket:[12958]
lrwx------ 1 root root 64 2011-12-05 10:54 149 -> socket:[12980]
lrwx------ 1 root root 64 2011-12-05 10:54 15 -> socket:[40001]
lrwx------ 1 root root 64 2011-12-05 10:54 150 -> socket:[14222]
lrwx------ 1 root root 64 2011-12-05 10:54 151 -> socket:[14259]
lrwx------ 1 root root 64 2011-12-05 10:54 152 -> socket:[14302]
lrwx------ 1 root root 64 2011-12-05 10:54 153 -> socket:[14313]
lrwx------ 1 root root 64 2011-12-05 10:54 154 -> socket:[14321]
lrwx------ 1 root root 64 2011-12-05 10:54 155 -> socket:[14334]
lrwx------ 1 root root 64 2011-12-05 10:54 156 -> socket:[14346]
lrwx------ 1 root root 64 2011-12-05 10:54 157 -> socket:[14348]
lrwx------ 1 root root 64 2011-12-05 10:54 158 -> socket:[14350]
lrwx------ 1 root root 64 2011-12-05 10:54 159 -> socket:[15519]
lrwx------ 1 root root 64 2011-12-05 10:54 16 -> socket:[39994]
lrwx------ 1 root root 64 2011-12-05 10:54 160 -> socket:[15668]
lrwx------ 1 root root 64 2011-12-05 10:54 161 -> socket:[15679]
lrwx------ 1 root root 64 2011-12-05 10:54 162 -> socket:[15727]
lrwx------ 1 root root 64 2011-12-05 10:54 163 -> socket:[15740]
lrwx------ 1 root root 64 2011-12-05 10:54 164 -> socket:[15757]
lrwx------ 1 root root 64 2011-12-05 10:54 165 -> socket:[15759]
lrwx------ 1 root root 64 2011-12-05 10:54 166 -> socket:[15761]
lrwx------ 1 root root 64 2011-12-05 10:54 167 -> socket:[15763]
lrwx------ 1 root root 64 2011-12-05 10:54 168 -> socket:[15765]
lrwx------ 1 root root 64 2011-12-05 10:54 169 -> socket:[15767]
lrwx------ 1 root root 64 2011-12-05 10:54 17 -> socket:[12488]
lrwx------ 1 root root 64 2011-12-05 10:54 170 -> socket:[15769]
lrwx------ 1 root root 64 2011-12-05 10:54 171 -> socket:[15771]
lrwx------ 1 root root 64 2011-12-05 10:54 172 -> socket:[15773]
lrwx------ 1 root root 64 2011-12-05 10:54 173 -> socket:[15775]
lrwx------ 1 root root 64 2011-12-05 10:54 174 -> socket:[15777]
lrwx------ 1 root root 64 2011-12-05 10:54 175 -> socket:[15779]
lrwx------ 1 root root 64 2011-12-05 10:54 176 -> socket:[15781]
lrwx------ 1 root root 64 2011-12-05 10:54 177 -> socket:[15783]
lrwx------ 1 root root 64 2011-12-05 10:54 178 -> socket:[15785]
lrwx------ 1 root root 64 2011-12-05 10:54 179 -> socket:[15795]
lrwx------ 1 root root 64 2011-12-05 10:54 18 -> socket:[10214]
lrwx------ 1 root root 64 2011-12-05 10:54 180 -> socket:[15796]
lrwx------ 1 root root 64 2011-12-05 10:54 181 -> socket:[15797]
lrwx------ 1 root root 64 2011-12-05 10:54 182 -> socket:[15798]
lrwx------ 1 root root 64 2011-12-05 10:54 183 -> socket:[15800]
lrwx------ 1 root root 64 2011-12-05 10:54 184 -> socket:[15802]
lrwx------ 1 root root 64 2011-12-05 10:54 185 -> socket:[15804]
lrwx------ 1 root root 64 2011-12-05 10:54 186 -> socket:[15806]
lrwx------ 1 root root 64 2011-12-05 10:54 187 -> socket:[15809]
lrwx------ 1 root root 64 2011-12-05 10:54 188 -> socket:[15810]
lrwx------ 1 root root 64 2011-12-05 10:54 189 -> socket:[15812]
lrwx------ 1 root root 64 2011-12-05 10:54 19 -> socket:[10656]
lrwx------ 1 root root 64 2011-12-05 10:54 190 -> socket:[15814]
lrwx------ 1 root root 64 2011-12-05 10:54 191 -> socket:[15816]
lrwx------ 1 root root 64 2011-12-05 10:54 192 -> socket:[15818]
lrwx------ 1 root root 64 2011-12-05 10:54 193 -> socket:[15821]
lrwx------ 1 root root 64 2011-12-05 10:54 194 -> socket:[15822]
lrwx------ 1 root root 64 2011-12-05 10:54 195 -> socket:[15824]
lrwx------ 1 root root 64 2011-12-05 10:54 196 -> socket:[15826]
lrwx------ 1 root root 64 2011-12-05 10:54 197 -> socket:[15828]
lrwx------ 1 root root 64 2011-12-05 10:54 198 -> socket:[15830]
lrwx------ 1 root root 64 2011-12-05 10:54 199 -> socket:[15852]
lr-x------ 1 root root 64 2011-12-05 10:54 2 -> /dev/null
lrwx------ 1 root root 64 2011-12-05 10:54 20 -> socket:[10847]
lrwx------ 1 root root 64 2011-12-05 10:54 200 -> socket:[15854]
lrwx------ 1 root root 64 2011-12-05 10:54 201 -> socket:[15856]
lrwx------ 1 root root 64 2011-12-05 10:54 202 -> socket:[15858]
lrwx------ 1 root root 64 2011-12-05 10:54 203 -> socket:[15860]
lrwx------ 1 root root 64 2011-12-05 10:54 204 -> socket:[15866]
lrwx------ 1 root root 64 2011-12-05 10:54 205 -> socket:[15873]
lrwx------ 1 root root 64 2011-12-05 10:54 206 -> socket:[15875]
lrwx------ 1 root root 64 2011-12-05 10:54 207 -> socket:[15877]
lrwx------ 1 root root 64 2011-12-05 10:54 208 -> socket:[15879]
lrwx------ 1 root root 64 2011-12-05 10:54 209 -> socket:[15881]
lrwx------ 1 root root 64 2011-12-05 10:54 21 -> socket:[10849]
lrwx------ 1 root root 64 2011-12-05 10:54 210 -> socket:[15895]
lrwx------ 1 root root 64 2011-12-05 10:54 211 -> socket:[15896]
lrwx------ 1 root root 64 2011-12-05 10:54 212 -> socket:[15897]
lrwx------ 1 root root 64 2011-12-05 10:54 213 -> socket:[15898]
lrwx------ 1 root root 64 2011-12-05 10:54 214 -> socket:[15899]
lrwx------ 1 root root 64 2011-12-05 10:54 215 -> socket:[15900]
lrwx------ 1 root root 64 2011-12-05 10:54 216 -> socket:[15901]
lrwx------ 1 root root 64 2011-12-05 10:54 217 -> socket:[15902]
lrwx------ 1 root root 64 2011-12-05 10:54 218 -> socket:[15904]
lrwx------ 1 root root 64 2011-12-05 10:54 219 -> socket:[15906]
lrwx------ 1 root root 64 2011-12-05 10:54 22 -> socket:[10856]
lrwx------ 1 root root 64 2011-12-05 10:54 220 -> socket:[15908]
lrwx------ 1 root root 64 2011-12-05 10:54 221 -> socket:[15910]
lrwx------ 1 root root 64 2011-12-05 10:54 222 -> socket:[15912]
lrwx------ 1 root root 64 2011-12-05 10:54 223 -> socket:[15914]
lrwx------ 1 root root 64 2011-12-05 10:54 224 -> socket:[15926]
lrwx------ 1 root root 64 2011-12-05 10:54 225 -> socket:[15928]
lrwx------ 1 root root 64 2011-12-05 10:54 226 -> socket:[15930]
lrwx------ 1 root root 64 2011-12-05 10:54 227 -> socket:[15932]
lrwx------ 1 root root 64 2011-12-05 10:54 228 -> socket:[15934]
lrwx------ 1 root root 64 2011-12-05 10:54 229 -> socket:[15936]
lrwx------ 1 root root 64 2011-12-05 10:54 23 -> socket:[10858]
lrwx------ 1 root root 64 2011-12-05 10:54 230 -> socket:[15943]
lrwx------ 1 root root 64 2011-12-05 10:54 231 -> socket:[15945]
lrwx------ 1 root root 64 2011-12-05 10:54 232 -> socket:[15947]
lrwx------ 1 root root 64 2011-12-05 10:54 233 -> socket:[15960]
lrwx------ 1 root root 64 2011-12-05 10:54 234 -> socket:[15961]
lrwx------ 1 root root 64 2011-12-05 10:54 235 -> socket:[15962]
lrwx------ 1 root root 64 2011-12-05 10:54 236 -> socket:[15963]
lrwx------ 1 root root 64 2011-12-05 10:54 237 -> socket:[15964]
lrwx------ 1 root root 64 2011-12-05 10:54 238 -> socket:[15965]
lrwx------ 1 root root 64 2011-12-05 10:54 239 -> socket:[15966]
lrwx------ 1 root root 64 2011-12-05 10:54 24 -> socket:[10863]
lrwx------ 1 root root 64 2011-12-05 10:54 240 -> socket:[15968]
lrwx------ 1 root root 64 2011-12-05 10:54 241 -> socket:[15970]
lrwx------ 1 root root 64 2011-12-05 10:54 242 -> socket:[15973]
lrwx------ 1 root root 64 2011-12-05 10:54 243 -> socket:[15974]
lrwx------ 1 root root 64 2011-12-05 10:54 244 -> socket:[15976]
lrwx------ 1 root root 64 2011-12-05 10:54 245 -> socket:[15984]
lrwx------ 1 root root 64 2011-12-05 10:54 246 -> socket:[15985]
lrwx------ 1 root root 64 2011-12-05 10:54 247 -> socket:[16002]
lrwx------ 1 root root 64 2011-12-05 10:54 248 -> socket:[16004]
lrwx------ 1 root root 64 2011-12-05 10:54 249 -> socket:[16006]
lrwx------ 1 root root 64 2011-12-05 10:54 25 -> socket:[10872]
lrwx------ 1 root root 64 2011-12-05 10:54 250 -> socket:[16008]
lrwx------ 1 root root 64 2011-12-05 10:54 251 -> socket:[16015]
lrwx------ 1 root root 64 2011-12-05 10:54 252 -> socket:[16017]
lrwx------ 1 root root 64 2011-12-05 10:54 253 -> socket:[16019]
lrwx------ 1 root root 64 2011-12-05 10:54 254 -> socket:[16021]
lrwx------ 1 root root 64 2011-12-05 10:54 255 -> socket:[16023]
lrwx------ 1 root root 64 2011-12-05 10:54 256 -> socket:[16035]
lrwx------ 1 root root 64 2011-12-05 10:54 257 -> socket:[16036]
lrwx------ 1 root root 64 2011-12-05 10:54 258 -> socket:[16037]
lrwx------ 1 root root 64 2011-12-05 10:54 259 -> socket:[16038]
lrwx------ 1 root root 64 2011-12-05 10:54 26 -> socket:[10873]
lrwx------ 1 root root 64 2011-12-05 10:54 260 -> socket:[16039]
lrwx------ 1 root root 64 2011-12-05 10:54 261 -> socket:[16040]
lrwx------ 1 root root 64 2011-12-05 10:54 262 -> socket:[16042]
lrwx------ 1 root root 64 2011-12-05 10:54 263 -> socket:[16044]
lrwx------ 1 root root 64 2011-12-05 10:54 264 -> socket:[16046]
lrwx------ 1 root root 64 2011-12-05 10:54 265 -> socket:[16048]
lrwx------ 1 root root 64 2011-12-05 10:54 266 -> socket:[16050]
lrwx------ 1 root root 64 2011-12-05 10:54 267 -> socket:[16057]
lrwx------ 1 root root 64 2011-12-05 10:54 268 -> socket:[16081]
lrwx------ 1 root root 64 2011-12-05 10:54 269 -> socket:[16083]
lrwx------ 1 root root 64 2011-12-05 10:54 27 -> socket:[10874]
lrwx------ 1 root root 64 2011-12-05 10:54 270 -> socket:[16095]
lrwx------ 1 root root 64 2011-12-05 10:54 271 -> socket:[16097]
lrwx------ 1 root root 64 2011-12-05 10:54 272 -> socket:[16099]
lrwx------ 1 root root 64 2011-12-05 10:54 273 -> socket:[16106]
lrwx------ 1 root root 64 2011-12-05 10:54 274 -> socket:[16108]
lrwx------ 1 root root 64 2011-12-05 10:54 275 -> socket:[16115]
lrwx------ 1 root root 64 2011-12-05 10:54 276 -> socket:[16117]
lrwx------ 1 root root 64 2011-12-05 10:54 277 -> socket:[16120]
lrwx------ 1 root root 64 2011-12-05 10:54 278 -> socket:[16121]
lrwx------ 1 root root 64 2011-12-05 10:54 279 -> socket:[16123]
lrwx------ 1 root root 64 2011-12-05 10:54 28 -> socket:[10876]
lrwx------ 1 root root 64 2011-12-05 10:54 280 -> socket:[16125]
lrwx------ 1 root root 64 2011-12-05 10:54 281 -> socket:[16128]
lrwx------ 1 root root 64 2011-12-05 10:54 282 -> socket:[16130]
lrwx------ 1 root root 64 2011-12-05 10:54 283 -> socket:[16131]
lrwx------ 1 root root 64 2011-12-05 10:54 284 -> socket:[16133]
lrwx------ 1 root root 64 2011-12-05 10:54 285 -> socket:[16135]
lrwx------ 1 root root 64 2011-12-05 10:54 286 -> socket:[16137]
lrwx------ 1 root root 64 2011-12-05 10:54 287 -> socket:[16149]
lrwx------ 1 root root 64 2011-12-05 10:54 288 -> socket:[16151]
lrwx------ 1 root root 64 2011-12-05 10:54 289 -> socket:[16153]
lrwx------ 1 root root 64 2011-12-05 10:54 29 -> socket:[10878]
lrwx------ 1 root root 64 2011-12-05 10:54 290 -> socket:[16161]
lrwx------ 1 root root 64 2011-12-05 10:54 291 -> socket:[16162]
lrwx------ 1 root root 64 2011-12-05 10:54 292 -> socket:[16174]
lrwx------ 1 root root 64 2011-12-05 10:54 293 -> socket:[16176]
lrwx------ 1 root root 64 2011-12-05 10:54 294 -> socket:[16183]
lrwx------ 1 root root 64 2011-12-05 10:54 295 -> socket:[16185]
lrwx------ 1 root root 64 2011-12-05 10:54 296 -> socket:[16187]
lrwx------ 1 root root 64 2011-12-05 10:54 297 -> socket:[16196]
lrwx------ 1 root root 64 2011-12-05 10:54 298 -> socket:[16197]
lrwx------ 1 root root 64 2011-12-05 10:54 299 -> socket:[16199]
l-wx------ 1 root root 64 2011-12-05 10:54 3 -> /var/log/ceph/mds.a.log
lrwx------ 1 root root 64 2011-12-05 10:54 30 -> socket:[10892]
lrwx------ 1 root root 64 2011-12-05 10:54 300 -> socket:[16200]
lrwx------ 1 root root 64 2011-12-05 10:54 301 -> socket:[16202]
lrwx------ 1 root root 64 2011-12-05 10:54 302 -> socket:[16204]
lrwx------ 1 root root 64 2011-12-05 10:54 303 -> socket:[16206]
lrwx------ 1 root root 64 2011-12-05 10:54 304 -> socket:[16213]
lrwx------ 1 root root 64 2011-12-05 10:54 305 -> socket:[16214]
lrwx------ 1 root root 64 2011-12-05 10:54 306 -> socket:[16216]
lrwx------ 1 root root 64 2011-12-05 10:54 307 -> socket:[16223]
lrwx------ 1 root root 64 2011-12-05 10:54 308 -> socket:[16225]
lrwx------ 1 root root 64 2011-12-05 10:54 309 -> socket:[16227]
lrwx------ 1 root root 64 2011-12-05 10:54 31 -> socket:[10919]
lrwx------ 1 root root 64 2011-12-05 10:54 310 -> socket:[16229]
lrwx------ 1 root root 64 2011-12-05 10:54 311 -> socket:[16231]
lrwx------ 1 root root 64 2011-12-05 10:54 312 -> socket:[16239]
lrwx------ 1 root root 64 2011-12-05 10:54 313 -> socket:[16240]
lrwx------ 1 root root 64 2011-12-05 10:54 314 -> socket:[16252]
lrwx------ 1 root root 64 2011-12-05 10:54 315 -> socket:[16265]
lrwx------ 1 root root 64 2011-12-05 10:54 316 -> socket:[16266]
lrwx------ 1 root root 64 2011-12-05 10:54 317 -> socket:[16273]
lrwx------ 1 root root 64 2011-12-05 10:54 318 -> socket:[16275]
lrwx------ 1 root root 64 2011-12-05 10:54 319 -> socket:[16282]
lrwx------ 1 root root 64 2011-12-05 10:54 32 -> socket:[39995]
lrwx------ 1 root root 64 2011-12-05 10:54 320 -> socket:[16284]
lrwx------ 1 root root 64 2011-12-05 10:54 321 -> socket:[16286]
lrwx------ 1 root root 64 2011-12-05 10:54 322 -> socket:[16288]
lrwx------ 1 root root 64 2011-12-05 10:54 323 -> socket:[16290]
lrwx------ 1 root root 64 2011-12-05 10:54 324 -> socket:[16297]
lrwx------ 1 root root 64 2011-12-05 10:54 325 -> socket:[16299]
lrwx------ 1 root root 64 2011-12-05 10:54 326 -> socket:[16301]
lrwx------ 1 root root 64 2011-12-05 10:54 327 -> socket:[16303]
lrwx------ 1 root root 64 2011-12-05 10:54 328 -> socket:[16305]
lrwx------ 1 root root 64 2011-12-05 10:54 329 -> socket:[16307]
lrwx------ 1 root root 64 2011-12-05 10:54 33 -> socket:[11572]
lrwx------ 1 root root 64 2011-12-05 10:54 330 -> socket:[16310]
lrwx------ 1 root root 64 2011-12-05 10:54 331 -> socket:[16311]
lrwx------ 1 root root 64 2011-12-05 10:54 332 -> socket:[16313]
lrwx------ 1 root root 64 2011-12-05 10:54 333 -> socket:[16315]
lrwx------ 1 root root 64 2011-12-05 10:54 334 -> socket:[16317]
lrwx------ 1 root root 64 2011-12-05 10:54 335 -> socket:[18794]
lrwx------ 1 root root 64 2011-12-05 10:54 336 -> socket:[16346]
lrwx------ 1 root root 64 2011-12-05 10:54 337 -> socket:[16358]
lrwx------ 1 root root 64 2011-12-05 10:54 338 -> socket:[16360]
lrwx------ 1 root root 64 2011-12-05 10:54 339 -> socket:[16362]
lrwx------ 1 root root 64 2011-12-05 10:54 34 -> socket:[12540]
lrwx------ 1 root root 64 2011-12-05 10:54 340 -> socket:[16370]
lrwx------ 1 root root 64 2011-12-05 10:54 341 -> socket:[16377]
lrwx------ 1 root root 64 2011-12-05 10:54 342 -> socket:[19386]
lrwx------ 1 root root 64 2011-12-05 10:54 343 -> socket:[19398]
lrwx------ 1 root root 64 2011-12-05 10:54 344 -> socket:[19435]
lrwx------ 1 root root 64 2011-12-05 10:54 345 -> socket:[19441]
lrwx------ 1 root root 64 2011-12-05 10:54 346 -> socket:[19489]
lrwx------ 1 root root 64 2011-12-05 10:54 347 -> socket:[19502]
lrwx------ 1 root root 64 2011-12-05 10:54 348 -> socket:[19519]
lrwx------ 1 root root 64 2011-12-05 10:54 349 -> socket:[19521]
lrwx------ 1 root root 64 2011-12-05 10:54 35 -> socket:[20392]
lrwx------ 1 root root 64 2011-12-05 10:54 350 -> socket:[19523]
lrwx------ 1 root root 64 2011-12-05 10:54 351 -> socket:[19525]
lrwx------ 1 root root 64 2011-12-05 10:54 352 -> socket:[19527]
lrwx------ 1 root root 64 2011-12-05 10:54 353 -> socket:[19529]
lrwx------ 1 root root 64 2011-12-05 10:54 354 -> socket:[19531]
lrwx------ 1 root root 64 2011-12-05 10:54 355 -> socket:[19533]
lrwx------ 1 root root 64 2011-12-05 10:54 356 -> socket:[19535]
lrwx------ 1 root root 64 2011-12-05 10:54 357 -> socket:[19537]
lrwx------ 1 root root 64 2011-12-05 10:54 358 -> socket:[19539]
lrwx------ 1 root root 64 2011-12-05 10:54 359 -> socket:[19541]
lrwx------ 1 root root 64 2011-12-05 10:54 36 -> socket:[39997]
lrwx------ 1 root root 64 2011-12-05 10:54 360 -> socket:[19543]
lrwx------ 1 root root 64 2011-12-05 10:54 361 -> socket:[19545]
lrwx------ 1 root root 64 2011-12-05 10:54 362 -> socket:[19547]
lrwx------ 1 root root 64 2011-12-05 10:54 363 -> socket:[19549]
lrwx------ 1 root root 64 2011-12-05 10:54 364 -> socket:[19551]
lrwx------ 1 root root 64 2011-12-05 10:54 365 -> socket:[19554]
lrwx------ 1 root root 64 2011-12-05 10:54 366 -> socket:[19555]
lrwx------ 1 root root 64 2011-12-05 10:54 367 -> socket:[19557]
lrwx------ 1 root root 64 2011-12-05 10:54 368 -> socket:[19559]
lrwx------ 1 root root 64 2011-12-05 10:54 369 -> socket:[19567]
lrwx------ 1 root root 64 2011-12-05 10:54 37 -> socket:[11612]
lrwx------ 1 root root 64 2011-12-05 10:54 370 -> socket:[19568]
lrwx------ 1 root root 64 2011-12-05 10:54 371 -> socket:[19571]
lrwx------ 1 root root 64 2011-12-05 10:54 372 -> socket:[19572]
lrwx------ 1 root root 64 2011-12-05 10:54 373 -> socket:[19575]
lrwx------ 1 root root 64 2011-12-05 10:54 374 -> socket:[19576]
lrwx------ 1 root root 64 2011-12-05 10:54 375 -> socket:[19578]
lrwx------ 1 root root 64 2011-12-05 10:54 376 -> socket:[19580]
lrwx------ 1 root root 64 2011-12-05 10:54 377 -> socket:[19583]
lrwx------ 1 root root 64 2011-12-05 10:54 378 -> socket:[19584]
lrwx------ 1 root root 64 2011-12-05 10:54 379 -> socket:[19586]
lrwx------ 1 root root 64 2011-12-05 10:54 38 -> socket:[11629]
lrwx------ 1 root root 64 2011-12-05 10:54 380 -> socket:[19588]
lrwx------ 1 root root 64 2011-12-05 10:54 381 -> socket:[19590]
lrwx------ 1 root root 64 2011-12-05 10:54 382 -> socket:[19592]
lrwx------ 1 root root 64 2011-12-05 10:54 383 -> socket:[19605]
lrwx------ 1 root root 64 2011-12-05 10:54 384 -> socket:[19607]
lrwx------ 1 root root 64 2011-12-05 10:54 385 -> socket:[19609]
lrwx------ 1 root root 64 2011-12-05 10:54 386 -> socket:[19611]
lrwx------ 1 root root 64 2011-12-05 10:54 387 -> socket:[19618]
lrwx------ 1 root root 64 2011-12-05 10:54 388 -> socket:[19620]
lrwx------ 1 root root 64 2011-12-05 10:54 389 -> socket:[19622]
lrwx------ 1 root root 64 2011-12-05 10:54 39 -> socket:[11689]
lrwx------ 1 root root 64 2011-12-05 10:54 390 -> socket:[19624]
lrwx------ 1 root root 64 2011-12-05 10:54 391 -> socket:[19626]
lrwx------ 1 root root 64 2011-12-05 10:54 392 -> socket:[19628]
lrwx------ 1 root root 64 2011-12-05 10:54 393 -> socket:[19630]
lrwx------ 1 root root 64 2011-12-05 10:54 394 -> socket:[19632]
lrwx------ 1 root root 64 2011-12-05 10:54 395 -> socket:[19641]
lrwx------ 1 root root 64 2011-12-05 10:54 396 -> socket:[19642]
lrwx------ 1 root root 64 2011-12-05 10:54 397 -> socket:[19643]
lrwx------ 1 root root 64 2011-12-05 10:54 398 -> socket:[19645]
lrwx------ 1 root root 64 2011-12-05 10:54 399 -> socket:[19647]
lrwx------ 1 root root 64 2011-12-05 10:54 4 -> socket:[7837]
lrwx------ 1 root root 64 2011-12-05 10:54 40 -> socket:[11716]
lrwx------ 1 root root 64 2011-12-05 10:54 400 -> socket:[19649]
lrwx------ 1 root root 64 2011-12-05 10:54 401 -> socket:[19651]
lrwx------ 1 root root 64 2011-12-05 10:54 402 -> socket:[19653]
lrwx------ 1 root root 64 2011-12-05 10:54 403 -> socket:[19666]
lrwx------ 1 root root 64 2011-12-05 10:54 404 -> socket:[19667]
lrwx------ 1 root root 64 2011-12-05 10:54 405 -> socket:[19668]
lrwx------ 1 root root 64 2011-12-05 10:54 406 -> socket:[19675]
lrwx------ 1 root root 64 2011-12-05 10:54 407 -> socket:[19677]
lrwx------ 1 root root 64 2011-12-05 10:54 408 -> socket:[19679]
lrwx------ 1 root root 64 2011-12-05 10:54 409 -> socket:[19681]
lrwx------ 1 root root 64 2011-12-05 10:54 41 -> socket:[11727]
lrwx------ 1 root root 64 2011-12-05 10:54 410 -> socket:[19683]
lrwx------ 1 root root 64 2011-12-05 10:54 411 -> socket:[19685]
lrwx------ 1 root root 64 2011-12-05 10:54 412 -> socket:[19687]
lrwx------ 1 root root 64 2011-12-05 10:54 413 -> socket:[19689]
lrwx------ 1 root root 64 2011-12-05 10:54 414 -> socket:[19691]
lrwx------ 1 root root 64 2011-12-05 10:54 415 -> socket:[19693]
lrwx------ 1 root root 64 2011-12-05 10:54 416 -> socket:[19695]
lrwx------ 1 root root 64 2011-12-05 10:54 417 -> socket:[19697]
lrwx------ 1 root root 64 2011-12-05 10:54 418 -> socket:[19699]
lrwx------ 1 root root 64 2011-12-05 10:54 419 -> socket:[19701]
lrwx------ 1 root root 64 2011-12-05 10:54 42 -> socket:[11748]
lrwx------ 1 root root 64 2011-12-05 10:54 420 -> socket:[19704]
lrwx------ 1 root root 64 2011-12-05 10:54 421 -> socket:[19705]
lrwx------ 1 root root 64 2011-12-05 10:54 422 -> socket:[19716]
lrwx------ 1 root root 64 2011-12-05 10:54 423 -> socket:[19718]
lrwx------ 1 root root 64 2011-12-05 10:54 424 -> socket:[19707]
lrwx------ 1 root root 64 2011-12-05 10:54 425 -> socket:[19709]
lrwx------ 1 root root 64 2011-12-05 10:54 426 -> socket:[19720]
lrwx------ 1 root root 64 2011-12-05 10:54 427 -> socket:[19727]
lrwx------ 1 root root 64 2011-12-05 10:54 428 -> socket:[19735]
lrwx------ 1 root root 64 2011-12-05 10:54 429 -> socket:[19736]
lrwx------ 1 root root 64 2011-12-05 10:54 43 -> socket:[11761]
lrwx------ 1 root root 64 2011-12-05 10:54 430 -> socket:[19738]
lrwx------ 1 root root 64 2011-12-05 10:54 431 -> socket:[19740]
lrwx------ 1 root root 64 2011-12-05 10:54 432 -> socket:[19742]
lrwx------ 1 root root 64 2011-12-05 10:54 433 -> socket:[19744]
lrwx------ 1 root root 64 2011-12-05 10:54 434 -> socket:[19746]
lrwx------ 1 root root 64 2011-12-05 10:54 435 -> socket:[19748]
lrwx------ 1 root root 64 2011-12-05 10:54 436 -> socket:[19750]
lrwx------ 1 root root 64 2011-12-05 10:54 437 -> socket:[19752]
lrwx------ 1 root root 64 2011-12-05 10:54 438 -> socket:[19754]
lrwx------ 1 root root 64 2011-12-05 10:54 439 -> socket:[19756]
lrwx------ 1 root root 64 2011-12-05 10:54 44 -> socket:[11778]
lrwx------ 1 root root 64 2011-12-05 10:54 440 -> socket:[19765]
lrwx------ 1 root root 64 2011-12-05 10:54 441 -> socket:[19766]
lrwx------ 1 root root 64 2011-12-05 10:54 442 -> socket:[19767]
lrwx------ 1 root root 64 2011-12-05 10:54 443 -> socket:[19769]
lrwx------ 1 root root 64 2011-12-05 10:54 444 -> socket:[19771]
lrwx------ 1 root root 64 2011-12-05 10:54 445 -> socket:[19773]
lrwx------ 1 root root 64 2011-12-05 10:54 446 -> socket:[19775]
lrwx------ 1 root root 64 2011-12-05 10:54 447 -> socket:[19777]
lrwx------ 1 root root 64 2011-12-05 10:54 448 -> socket:[19784]
lrwx------ 1 root root 64 2011-12-05 10:54 449 -> socket:[19786]
lrwx------ 1 root root 64 2011-12-05 10:54 45 -> socket:[11780]
lrwx------ 1 root root 64 2011-12-05 10:54 450 -> socket:[19788]
lrwx------ 1 root root 64 2011-12-05 10:54 451 -> socket:[19795]
lrwx------ 1 root root 64 2011-12-05 10:54 452 -> socket:[19797]
lrwx------ 1 root root 64 2011-12-05 10:54 453 -> socket:[19799]
lrwx------ 1 root root 64 2011-12-05 10:54 454 -> socket:[19801]
lrwx------ 1 root root 64 2011-12-05 10:54 455 -> socket:[19803]
lrwx------ 1 root root 64 2011-12-05 10:54 456 -> socket:[19811]
lrwx------ 1 root root 64 2011-12-05 10:54 457 -> socket:[19812]
lrwx------ 1 root root 64 2011-12-05 10:54 458 -> socket:[19814]
lrwx------ 1 root root 64 2011-12-05 10:54 459 -> socket:[19816]
lrwx------ 1 root root 64 2011-12-05 10:54 46 -> socket:[11787]
lrwx------ 1 root root 64 2011-12-05 10:54 460 -> socket:[19819]
lrwx------ 1 root root 64 2011-12-05 10:54 461 -> socket:[19820]
lrwx------ 1 root root 64 2011-12-05 10:54 462 -> socket:[19830]
lrwx------ 1 root root 64 2011-12-05 10:54 463 -> socket:[19831]
lrwx------ 1 root root 64 2011-12-05 10:54 464 -> socket:[19832]
lrwx------ 1 root root 64 2011-12-05 10:54 465 -> socket:[19833]
lrwx------ 1 root root 64 2011-12-05 10:54 466 -> socket:[19835]
lrwx------ 1 root root 64 2011-12-05 10:54 467 -> socket:[19837]
lrwx------ 1 root root 64 2011-12-05 10:54 468 -> socket:[19848]
lrwx------ 1 root root 64 2011-12-05 10:54 469 -> socket:[19849]
lrwx------ 1 root root 64 2011-12-05 10:54 47 -> socket:[40000]
lrwx------ 1 root root 64 2011-12-05 10:54 470 -> socket:[19850]
lrwx------ 1 root root 64 2011-12-05 10:54 471 -> socket:[19851]
lrwx------ 1 root root 64 2011-12-05 10:54 472 -> socket:[19852]
lrwx------ 1 root root 64 2011-12-05 10:54 473 -> socket:[19854]
lrwx------ 1 root root 64 2011-12-05 10:54 474 -> socket:[19856]
lrwx------ 1 root root 64 2011-12-05 10:54 475 -> socket:[19863]
lrwx------ 1 root root 64 2011-12-05 10:54 476 -> socket:[19865]
lrwx------ 1 root root 64 2011-12-05 10:54 477 -> socket:[19867]
lrwx------ 1 root root 64 2011-12-05 10:54 478 -> socket:[19869]
lrwx------ 1 root root 64 2011-12-05 10:54 479 -> socket:[19871]
lrwx------ 1 root root 64 2011-12-05 10:54 48 -> socket:[39996]
lrwx------ 1 root root 64 2011-12-05 10:54 480 -> socket:[19878]
lrwx------ 1 root root 64 2011-12-05 10:54 481 -> socket:[19880]
lrwx------ 1 root root 64 2011-12-05 10:54 482 -> socket:[19882]
lrwx------ 1 root root 64 2011-12-05 10:54 483 -> socket:[19884]
lrwx------ 1 root root 64 2011-12-05 10:54 484 -> socket:[19886]
lrwx------ 1 root root 64 2011-12-05 10:54 485 -> socket:[19888]
lrwx------ 1 root root 64 2011-12-05 10:54 486 -> socket:[19890]
lrwx------ 1 root root 64 2011-12-05 10:54 487 -> socket:[19892]
lrwx------ 1 root root 64 2011-12-05 10:54 488 -> socket:[19894]
lrwx------ 1 root root 64 2011-12-05 10:54 489 -> socket:[19902]
lrwx------ 1 root root 64 2011-12-05 10:54 49 -> socket:[12551]
lrwx------ 1 root root 64 2011-12-05 10:54 490 -> socket:[19903]
lrwx------ 1 root root 64 2011-12-05 10:54 491 -> socket:[19905]
lrwx------ 1 root root 64 2011-12-05 10:54 492 -> socket:[19907]
lrwx------ 1 root root 64 2011-12-05 10:54 493 -> socket:[19909]
lrwx------ 1 root root 64 2011-12-05 10:54 494 -> socket:[19911]
lrwx------ 1 root root 64 2011-12-05 10:54 495 -> socket:[19913]
lrwx------ 1 root root 64 2011-12-05 10:54 496 -> socket:[19915]
lrwx------ 1 root root 64 2011-12-05 10:54 497 -> socket:[19922]
lrwx------ 1 root root 64 2011-12-05 10:54 498 -> socket:[19924]
lrwx------ 1 root root 64 2011-12-05 10:54 499 -> socket:[19926]
lr-x------ 1 root root 64 2011-12-05 10:54 5 -> pipe:[7841]
lrwx------ 1 root root 64 2011-12-05 10:54 50 -> socket:[12605]
lrwx------ 1 root root 64 2011-12-05 10:54 500 -> socket:[19934]
lrwx------ 1 root root 64 2011-12-05 10:54 501 -> socket:[19935]
lrwx------ 1 root root 64 2011-12-05 10:54 502 -> socket:[19937]
lrwx------ 1 root root 64 2011-12-05 10:54 503 -> socket:[19943]
lrwx------ 1 root root 64 2011-12-05 10:54 504 -> socket:[19951]
lrwx------ 1 root root 64 2011-12-05 10:54 505 -> socket:[19952]
lrwx------ 1 root root 64 2011-12-05 10:54 506 -> socket:[19954]
lrwx------ 1 root root 64 2011-12-05 10:54 507 -> socket:[19956]
lrwx------ 1 root root 64 2011-12-05 10:54 508 -> socket:[19958]
lrwx------ 1 root root 64 2011-12-05 10:54 509 -> socket:[19960]
lrwx------ 1 root root 64 2011-12-05 10:54 51 -> socket:[12618]
lrwx------ 1 root root 64 2011-12-05 10:54 510 -> socket:[19962]
lrwx------ 1 root root 64 2011-12-05 10:54 511 -> socket:[40003]
lrwx------ 1 root root 64 2011-12-05 10:54 512 -> socket:[19974]
lrwx------ 1 root root 64 2011-12-05 10:54 513 -> socket:[19975]
lrwx------ 1 root root 64 2011-12-05 10:54 514 -> socket:[19976]
lrwx------ 1 root root 64 2011-12-05 10:54 515 -> socket:[19977]
lrwx------ 1 root root 64 2011-12-05 10:54 516 -> socket:[20409]
lrwx------ 1 root root 64 2011-12-05 10:54 517 -> socket:[19986]
lrwx------ 1 root root 64 2011-12-05 10:54 518 -> socket:[19988]
lrwx------ 1 root root 64 2011-12-05 10:54 519 -> socket:[20446]
lrwx------ 1 root root 64 2011-12-05 10:54 52 -> socket:[12630]
lrwx------ 1 root root 64 2011-12-05 10:54 520 -> socket:[19997]
lrwx------ 1 root root 64 2011-12-05 10:54 521 -> socket:[19999]
lrwx------ 1 root root 64 2011-12-05 10:54 522 -> socket:[20001]
lrwx------ 1 root root 64 2011-12-05 10:54 523 -> socket:[20003]
lrwx------ 1 root root 64 2011-12-05 10:54 524 -> socket:[20010]
lrwx------ 1 root root 64 2011-12-05 10:54 525 -> socket:[20012]
lrwx------ 1 root root 64 2011-12-05 10:54 526 -> socket:[20014]
lrwx------ 1 root root 64 2011-12-05 10:54 527 -> socket:[20022]
lrwx------ 1 root root 64 2011-12-05 10:54 528 -> socket:[20023]
lrwx------ 1 root root 64 2011-12-05 10:54 529 -> socket:[20025]
lrwx------ 1 root root 64 2011-12-05 10:54 53 -> socket:[12632]
lrwx------ 1 root root 64 2011-12-05 10:54 530 -> socket:[20032]
lrwx------ 1 root root 64 2011-12-05 10:54 531 -> socket:[20039]
lrwx------ 1 root root 64 2011-12-05 10:54 532 -> socket:[20046]
lrwx------ 1 root root 64 2011-12-05 10:54 533 -> socket:[20110]
lrwx------ 1 root root 64 2011-12-05 10:54 534 -> socket:[20134]
lrwx------ 1 root root 64 2011-12-05 10:54 535 -> socket:[20146]
lrwx------ 1 root root 64 2011-12-05 10:54 536 -> socket:[20158]
lrwx------ 1 root root 64 2011-12-05 10:54 537 -> socket:[20170]
lrwx------ 1 root root 64 2011-12-05 10:54 538 -> socket:[20192]
lrwx------ 1 root root 64 2011-12-05 10:54 54 -> socket:[12634]
lrwx------ 1 root root 64 2011-12-05 10:54 55 -> socket:[12636]
lrwx------ 1 root root 64 2011-12-05 10:54 56 -> socket:[12639]
lrwx------ 1 root root 64 2011-12-05 10:54 57 -> socket:[12640]
lrwx------ 1 root root 64 2011-12-05 10:54 58 -> socket:[12642]
lrwx------ 1 root root 64 2011-12-05 10:54 59 -> socket:[12644]
l-wx------ 1 root root 64 2011-12-05 10:54 6 -> pipe:[7841]
lrwx------ 1 root root 64 2011-12-05 10:54 60 -> socket:[12646]
lrwx------ 1 root root 64 2011-12-05 10:54 61 -> socket:[12648]
lrwx------ 1 root root 64 2011-12-05 10:54 62 -> socket:[12651]
lrwx------ 1 root root 64 2011-12-05 10:54 63 -> socket:[12652]
lrwx------ 1 root root 64 2011-12-05 10:54 64 -> socket:[12654]
lrwx------ 1 root root 64 2011-12-05 10:54 65 -> socket:[12656]
lrwx------ 1 root root 64 2011-12-05 10:54 66 -> socket:[12659]
lrwx------ 1 root root 64 2011-12-05 10:54 67 -> socket:[12660]
lrwx------ 1 root root 64 2011-12-05 10:54 68 -> socket:[12662]
lrwx------ 1 root root 64 2011-12-05 10:54 69 -> socket:[12664]
lrwx------ 1 root root 64 2011-12-05 10:54 7 -> socket:[7842]
lrwx------ 1 root root 64 2011-12-05 10:54 70 -> socket:[12666]
lrwx------ 1 root root 64 2011-12-05 10:54 71 -> socket:[12674]
lrwx------ 1 root root 64 2011-12-05 10:54 72 -> socket:[12675]
lrwx------ 1 root root 64 2011-12-05 10:54 73 -> socket:[12682]
lrwx------ 1 root root 64 2011-12-05 10:54 74 -> socket:[12684]
lrwx------ 1 root root 64 2011-12-05 10:54 75 -> socket:[12687]
lrwx------ 1 root root 64 2011-12-05 10:54 76 -> socket:[12688]
lrwx------ 1 root root 64 2011-12-05 10:54 77 -> socket:[12691]
lrwx------ 1 root root 64 2011-12-05 10:54 78 -> socket:[12692]
lrwx------ 1 root root 64 2011-12-05 10:54 79 -> socket:[12694]
lrwx------ 1 root root 64 2011-12-05 10:54 8 -> socket:[7850]
lrwx------ 1 root root 64 2011-12-05 10:54 80 -> socket:[12696]
lrwx------ 1 root root 64 2011-12-05 10:54 81 -> socket:[12698]
lrwx------ 1 root root 64 2011-12-05 10:54 82 -> socket:[12700]
lrwx------ 1 root root 64 2011-12-05 10:54 83 -> socket:[12702]
lrwx------ 1 root root 64 2011-12-05 10:54 84 -> socket:[12704]
lrwx------ 1 root root 64 2011-12-05 10:54 85 -> socket:[12706]
lrwx------ 1 root root 64 2011-12-05 10:54 86 -> socket:[12708]
lrwx------ 1 root root 64 2011-12-05 10:54 87 -> socket:[12732]
lrwx------ 1 root root 64 2011-12-05 10:54 88 -> socket:[12734]
lrwx------ 1 root root 64 2011-12-05 10:54 89 -> socket:[12746]
lrwx------ 1 root root 64 2011-12-05 10:54 9 -> socket:[39998]
lrwx------ 1 root root 64 2011-12-05 10:54 90 -> socket:[12748]
lrwx------ 1 root root 64 2011-12-05 10:54 91 -> socket:[12750]
lrwx------ 1 root root 64 2011-12-05 10:54 92 -> socket:[12752]
lrwx------ 1 root root 64 2011-12-05 10:54 93 -> socket:[12754]
lrwx------ 1 root root 64 2011-12-05 10:54 94 -> socket:[12756]
lrwx------ 1 root root 64 2011-12-05 10:54 95 -> socket:[12758]
lrwx------ 1 root root 64 2011-12-05 10:54 96 -> socket:[12760]
lrwx------ 1 root root 64 2011-12-05 10:54 97 -> socket:[12762]
lrwx------ 1 root root 64 2011-12-05 10:54 98 -> socket:[12764]
lrwx------ 1 root root 64 2011-12-05 10:54 99 -> socket:[12766]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 18:57 ` Noah Watkins
@ 2011-12-05 19:03 ` Tommi Virtanen
2011-12-05 19:05 ` Noah Watkins
0 siblings, 1 reply; 12+ messages in thread
From: Tommi Virtanen @ 2011-12-05 19:03 UTC (permalink / raw)
To: Noah Watkins; +Cc: Sage Weil, Noah Watkins, ceph-devel
On Mon, Dec 5, 2011 at 10:57, Noah Watkins <jayhawk@cs.ucsc.edu> wrote:
> Here is the proc/<mds_pid>/fd contents. These contents should reflect an
> idle cluster with no active clients.
...
> root@issdm-23:~# ls -la /proc/1394/fd/
> lrwx------ 1 root root 64 2011-12-05 10:54 10 -> socket:[40002]
Can we get a "sudo lsof -n -p 1394" too? That would tell a little bit
more about what the sockets are actually for..
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 19:03 ` Tommi Virtanen
@ 2011-12-05 19:05 ` Noah Watkins
2011-12-05 19:11 ` Sage Weil
2011-12-05 19:36 ` Tommi Virtanen
0 siblings, 2 replies; 12+ messages in thread
From: Noah Watkins @ 2011-12-05 19:05 UTC (permalink / raw)
To: Tommi Virtanen; +Cc: Noah Watkins, Sage Weil, ceph-devel
On 12/05/2011 11:03 AM, Tommi Virtanen wrote:
> Can we get a "sudo lsof -n -p 1394" too? That would tell a little bit
> more about what the sockets are actually for..
>
root@issdm-23:~# lsof -n -p 1394
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ceph-mds 1394 root cwd DIR 8,1 4096 2 /
ceph-mds 1394 root rtd DIR 8,1 4096 2 /
ceph-mds 1394 root txt REG 8,1 4577640 7097096
/usr/bin/ceph-mds
ceph-mds 1394 root mem REG 8,1 42856 7096319
/usr/lib/libunwind.so.7.0.0
ceph-mds 1394 root mem REG 8,1 1572232 9178738
/lib/libc-2.12.1.so
ceph-mds 1394 root mem REG 8,1 88384 9175100
/lib/libgcc_s.so.1
ceph-mds 1394 root mem REG 8,1 534832 9178742
/lib/libm-2.12.1.so
ceph-mds 1394 root mem REG 8,1 986888 7082428
/usr/lib/libstdc++.so.6.0.14
ceph-mds 1394 root mem REG 8,1 249080 7096323
/usr/lib/libtcmalloc.so.0.0.0
ceph-mds 1394 root mem REG 8,1 31744 9178754
/lib/librt-2.12.1.so
ceph-mds 1394 root mem REG 8,1 19008 9175163
/lib/libuuid.so.1.3.0
ceph-mds 1394 root mem REG 8,1 4946936 7096152
/usr/lib/libcrypto++.so.8.0.0
ceph-mds 1394 root mem REG 8,1 136067 9178752
/lib/libpthread-2.12.1.so
ceph-mds 1394 root mem REG 8,1 141072 9178735
/lib/ld-2.12.1.so
ceph-mds 1394 root 0r CHR 1,3 0t0 4372 /dev/null
ceph-mds 1394 root 1r CHR 1,3 0t0 4372 /dev/null
ceph-mds 1394 root 2r CHR 1,3 0t0 4372 /dev/null
ceph-mds 1394 root 3w REG 8,1 14761050 6034073
/var/log/ceph/mds.a.log
ceph-mds 1394 root 4u IPv4 7837 0t0 TCP
*:6800 (LISTEN)
ceph-mds 1394 root 5r FIFO 0,8 0t0 7841 pipe
ceph-mds 1394 root 6w FIFO 0,8 0t0 7841 pipe
ceph-mds 1394 root 7u unix 0xffff88021a623a80 0t0 7842
/var/run/ceph/mds.a.asok
ceph-mds 1394 root 8u IPv4 7850 0t0 TCP
192.168.141.123:38046->192.168.141.123:6789 (ESTABLISHED)
ceph-mds 1394 root 9u IPv4 41245 0t0 TCP
192.168.141.123:58242->192.168.141.135:6800 (ESTABLISHED)
ceph-mds 1394 root 10u IPv4 41249 0t0 TCP
192.168.141.123:39475->192.168.141.128:6800 (ESTABLISHED)
ceph-mds 1394 root 11u IPv4 40004 0t0 TCP
192.168.141.123:53885->192.168.141.124:6800 (ESTABLISHED)
ceph-mds 1394 root 12u sock 0,6 0t0 11560 can't
identify protocol
ceph-mds 1394 root 13u IPv4 41246 0t0 TCP
192.168.141.123:51006->192.168.141.132:6800 (ESTABLISHED)
ceph-mds 1394 root 14u IPv4 40005 0t0 TCP
192.168.141.123:34639->192.168.141.138:6800 (ESTABLISHED)
ceph-mds 1394 root 15u IPv4 41247 0t0 TCP
192.168.141.123:43558->192.168.141.125:6800 (ESTABLISHED)
ceph-mds 1394 root 16u IPv4 41241 0t0 TCP
192.168.141.123:52571->192.168.141.127:6800 (ESTABLISHED)
ceph-mds 1394 root 17u sock 0,6 0t0 12488 can't
identify protocol
ceph-mds 1394 root 18u sock 0,6 0t0 10214 can't
identify protocol
ceph-mds 1394 root 19u IPv4 10656 0t0 TCP
192.168.141.123:6800->192.168.141.123:46078 (ESTABLISHED)
ceph-mds 1394 root 20u IPv4 10847 0t0 TCP
192.168.141.123:6800->192.168.141.130:37586 (ESTABLISHED)
ceph-mds 1394 root 21u IPv4 10849 0t0 TCP
192.168.141.123:6800->192.168.141.135:44058 (ESTABLISHED)
ceph-mds 1394 root 22u IPv4 10856 0t0 TCP
192.168.141.123:6800->192.168.141.129:34711 (ESTABLISHED)
ceph-mds 1394 root 23u IPv4 10858 0t0 TCP
192.168.141.123:6800->192.168.141.137:35032 (ESTABLISHED)
ceph-mds 1394 root 24u IPv4 10863 0t0 TCP
192.168.141.123:6800->192.168.141.131:34520 (ESTABLISHED)
ceph-mds 1394 root 25u IPv4 10872 0t0 TCP
192.168.141.123:6800->192.168.141.122:44304 (ESTABLISHED)
ceph-mds 1394 root 26u IPv4 10873 0t0 TCP
192.168.141.123:6800->192.168.141.128:51902 (ESTABLISHED)
ceph-mds 1394 root 27u IPv4 10874 0t0 TCP
192.168.141.123:6800->192.168.141.125:60607 (ESTABLISHED)
ceph-mds 1394 root 28u IPv4 10876 0t0 TCP
192.168.141.123:6800->192.168.141.124:39850 (ESTABLISHED)
ceph-mds 1394 root 29u IPv4 10878 0t0 TCP
192.168.141.123:6800->192.168.141.127:43072 (ESTABLISHED)
ceph-mds 1394 root 30u IPv4 10892 0t0 TCP
192.168.141.123:6800->192.168.141.132:41561 (ESTABLISHED)
ceph-mds 1394 root 31u IPv4 10919 0t0 TCP
192.168.141.123:6800->192.168.141.138:36645 (ESTABLISHED)
ceph-mds 1394 root 32u IPv4 41242 0t0 TCP
192.168.141.123:56058->192.168.141.137:6800 (ESTABLISHED)
ceph-mds 1394 root 33u sock 0,6 0t0 11572 can't
identify protocol
ceph-mds 1394 root 34u sock 0,6 0t0 12540 can't
identify protocol
ceph-mds 1394 root 35u sock 0,6 0t0 20392 can't
identify protocol
ceph-mds 1394 root 36u IPv4 41244 0t0 TCP
192.168.141.123:47859->192.168.141.122:6800 (ESTABLISHED)
ceph-mds 1394 root 37u sock 0,6 0t0 11612 can't
identify protocol
ceph-mds 1394 root 38u sock 0,6 0t0 11629 can't
identify protocol
ceph-mds 1394 root 39u sock 0,6 0t0 11689 can't
identify protocol
ceph-mds 1394 root 40u sock 0,6 0t0 11716 can't
identify protocol
ceph-mds 1394 root 41u sock 0,6 0t0 11727 can't
identify protocol
ceph-mds 1394 root 42u sock 0,6 0t0 11748 can't
identify protocol
ceph-mds 1394 root 43u sock 0,6 0t0 11761 can't
identify protocol
ceph-mds 1394 root 44u sock 0,6 0t0 11778 can't
identify protocol
ceph-mds 1394 root 45u sock 0,6 0t0 11780 can't
identify protocol
ceph-mds 1394 root 46u sock 0,6 0t0 11787 can't
identify protocol
ceph-mds 1394 root 47u IPv4 41248 0t0 TCP
192.168.141.123:54637->192.168.141.131:6800 (ESTABLISHED)
ceph-mds 1394 root 48u IPv4 41243 0t0 TCP
192.168.141.123:48293->192.168.141.130:6800 (ESTABLISHED)
ceph-mds 1394 root 49u sock 0,6 0t0 12551 can't
identify protocol
ceph-mds 1394 root 50u sock 0,6 0t0 12605 can't
identify protocol
ceph-mds 1394 root 51u sock 0,6 0t0 12618 can't
identify protocol
ceph-mds 1394 root 52u sock 0,6 0t0 12630 can't
identify protocol
ceph-mds 1394 root 53u sock 0,6 0t0 12632 can't
identify protocol
ceph-mds 1394 root 54u sock 0,6 0t0 12634 can't
identify protocol
ceph-mds 1394 root 55u sock 0,6 0t0 12636 can't
identify protocol
ceph-mds 1394 root 56u sock 0,6 0t0 12639 can't
identify protocol
ceph-mds 1394 root 57u sock 0,6 0t0 12640 can't
identify protocol
ceph-mds 1394 root 58u sock 0,6 0t0 12642 can't
identify protocol
ceph-mds 1394 root 59u sock 0,6 0t0 12644 can't
identify protocol
ceph-mds 1394 root 60u sock 0,6 0t0 12646 can't
identify protocol
ceph-mds 1394 root 61u sock 0,6 0t0 12648 can't
identify protocol
ceph-mds 1394 root 62u sock 0,6 0t0 12651 can't
identify protocol
ceph-mds 1394 root 63u sock 0,6 0t0 12652 can't
identify protocol
ceph-mds 1394 root 64u sock 0,6 0t0 12654 can't
identify protocol
ceph-mds 1394 root 65u sock 0,6 0t0 12656 can't
identify protocol
ceph-mds 1394 root 66u sock 0,6 0t0 12659 can't
identify protocol
ceph-mds 1394 root 67u sock 0,6 0t0 12660 can't
identify protocol
ceph-mds 1394 root 68u sock 0,6 0t0 12662 can't
identify protocol
ceph-mds 1394 root 69u sock 0,6 0t0 12664 can't
identify protocol
ceph-mds 1394 root 70u sock 0,6 0t0 12666 can't
identify protocol
ceph-mds 1394 root 71u sock 0,6 0t0 12674 can't
identify protocol
ceph-mds 1394 root 72u sock 0,6 0t0 12675 can't
identify protocol
ceph-mds 1394 root 73u sock 0,6 0t0 12682 can't
identify protocol
ceph-mds 1394 root 74u sock 0,6 0t0 12684 can't
identify protocol
ceph-mds 1394 root 75u sock 0,6 0t0 12687 can't
identify protocol
ceph-mds 1394 root 76u sock 0,6 0t0 12688 can't
identify protocol
ceph-mds 1394 root 77u sock 0,6 0t0 12691 can't
identify protocol
ceph-mds 1394 root 78u sock 0,6 0t0 12692 can't
identify protocol
ceph-mds 1394 root 79u sock 0,6 0t0 12694 can't
identify protocol
ceph-mds 1394 root 80u sock 0,6 0t0 12696 can't
identify protocol
ceph-mds 1394 root 81u sock 0,6 0t0 12698 can't
identify protocol
ceph-mds 1394 root 82u sock 0,6 0t0 12700 can't
identify protocol
ceph-mds 1394 root 83u sock 0,6 0t0 12702 can't
identify protocol
ceph-mds 1394 root 84u sock 0,6 0t0 12704 can't
identify protocol
ceph-mds 1394 root 85u sock 0,6 0t0 12706 can't
identify protocol
ceph-mds 1394 root 86u sock 0,6 0t0 12708 can't
identify protocol
ceph-mds 1394 root 87u sock 0,6 0t0 12732 can't
identify protocol
ceph-mds 1394 root 88u sock 0,6 0t0 12734 can't
identify protocol
ceph-mds 1394 root 89u sock 0,6 0t0 12746 can't
identify protocol
ceph-mds 1394 root 90u sock 0,6 0t0 12748 can't
identify protocol
ceph-mds 1394 root 91u sock 0,6 0t0 12750 can't
identify protocol
ceph-mds 1394 root 92u sock 0,6 0t0 12752 can't
identify protocol
ceph-mds 1394 root 93u sock 0,6 0t0 12754 can't
identify protocol
ceph-mds 1394 root 94u sock 0,6 0t0 12756 can't
identify protocol
ceph-mds 1394 root 95u sock 0,6 0t0 12758 can't
identify protocol
ceph-mds 1394 root 96u sock 0,6 0t0 12760 can't
identify protocol
ceph-mds 1394 root 97u sock 0,6 0t0 12762 can't
identify protocol
ceph-mds 1394 root 98u sock 0,6 0t0 12764 can't
identify protocol
ceph-mds 1394 root 99u sock 0,6 0t0 12766 can't
identify protocol
ceph-mds 1394 root 100u sock 0,6 0t0 12769 can't
identify protocol
ceph-mds 1394 root 101u sock 0,6 0t0 12770 can't
identify protocol
ceph-mds 1394 root 102u sock 0,6 0t0 12773 can't
identify protocol
ceph-mds 1394 root 103u sock 0,6 0t0 12774 can't
identify protocol
ceph-mds 1394 root 104u sock 0,6 0t0 12776 can't
identify protocol
ceph-mds 1394 root 105u sock 0,6 0t0 12786 can't
identify protocol
ceph-mds 1394 root 106u sock 0,6 0t0 12787 can't
identify protocol
ceph-mds 1394 root 107u sock 0,6 0t0 12788 can't
identify protocol
ceph-mds 1394 root 108u sock 0,6 0t0 12789 can't
identify protocol
ceph-mds 1394 root 109u sock 0,6 0t0 12790 can't
identify protocol
ceph-mds 1394 root 110u sock 0,6 0t0 12803 can't
identify protocol
ceph-mds 1394 root 111u sock 0,6 0t0 12804 can't
identify protocol
ceph-mds 1394 root 112u sock 0,6 0t0 12816 can't
identify protocol
ceph-mds 1394 root 113u sock 0,6 0t0 12818 can't
identify protocol
ceph-mds 1394 root 114u sock 0,6 0t0 12831 can't
identify protocol
ceph-mds 1394 root 115u sock 0,6 0t0 12832 can't
identify protocol
ceph-mds 1394 root 116u sock 0,6 0t0 12834 can't
identify protocol
ceph-mds 1394 root 117u sock 0,6 0t0 12836 can't
identify protocol
ceph-mds 1394 root 118u sock 0,6 0t0 12838 can't
identify protocol
ceph-mds 1394 root 119u sock 0,6 0t0 12840 can't
identify protocol
ceph-mds 1394 root 120u sock 0,6 0t0 12842 can't
identify protocol
ceph-mds 1394 root 121u sock 0,6 0t0 12844 can't
identify protocol
ceph-mds 1394 root 122u sock 0,6 0t0 12846 can't
identify protocol
ceph-mds 1394 root 123u sock 0,6 0t0 12848 can't
identify protocol
ceph-mds 1394 root 124u sock 0,6 0t0 12851 can't
identify protocol
ceph-mds 1394 root 125u sock 0,6 0t0 12852 can't
identify protocol
ceph-mds 1394 root 126u sock 0,6 0t0 12854 can't
identify protocol
ceph-mds 1394 root 127u sock 0,6 0t0 12856 can't
identify protocol
ceph-mds 1394 root 128u sock 0,6 0t0 12858 can't
identify protocol
ceph-mds 1394 root 129u sock 0,6 0t0 12860 can't
identify protocol
ceph-mds 1394 root 130u sock 0,6 0t0 12862 can't
identify protocol
ceph-mds 1394 root 131u sock 0,6 0t0 12864 can't
identify protocol
ceph-mds 1394 root 132u sock 0,6 0t0 12866 can't
identify protocol
ceph-mds 1394 root 133u sock 0,6 0t0 12873 can't
identify protocol
ceph-mds 1394 root 134u sock 0,6 0t0 12880 can't
identify protocol
ceph-mds 1394 root 135u sock 0,6 0t0 12882 can't
identify protocol
ceph-mds 1394 root 136u sock 0,6 0t0 12894 can't
identify protocol
ceph-mds 1394 root 137u sock 0,6 0t0 12896 can't
identify protocol
ceph-mds 1394 root 138u sock 0,6 0t0 12908 can't
identify protocol
ceph-mds 1394 root 139u sock 0,6 0t0 12910 can't
identify protocol
ceph-mds 1394 root 140u sock 0,6 0t0 14170 can't
identify protocol
ceph-mds 1394 root 141u sock 0,6 0t0 12915 can't
identify protocol
ceph-mds 1394 root 142u sock 0,6 0t0 12916 can't
identify protocol
ceph-mds 1394 root 143u sock 0,6 0t0 12918 can't
identify protocol
ceph-mds 1394 root 144u sock 0,6 0t0 12920 can't
identify protocol
ceph-mds 1394 root 145u sock 0,6 0t0 12927 can't
identify protocol
ceph-mds 1394 root 146u sock 0,6 0t0 12944 can't
identify protocol
ceph-mds 1394 root 147u sock 0,6 0t0 12956 can't
identify protocol
ceph-mds 1394 root 148u sock 0,6 0t0 12958 can't
identify protocol
ceph-mds 1394 root 149u sock 0,6 0t0 12980 can't
identify protocol
ceph-mds 1394 root 150u sock 0,6 0t0 14222 can't
identify protocol
ceph-mds 1394 root 151u sock 0,6 0t0 14259 can't
identify protocol
ceph-mds 1394 root 152u sock 0,6 0t0 14302 can't
identify protocol
ceph-mds 1394 root 153u sock 0,6 0t0 14313 can't
identify protocol
ceph-mds 1394 root 154u sock 0,6 0t0 14321 can't
identify protocol
ceph-mds 1394 root 155u sock 0,6 0t0 14334 can't
identify protocol
ceph-mds 1394 root 156u sock 0,6 0t0 14346 can't
identify protocol
ceph-mds 1394 root 157u sock 0,6 0t0 14348 can't
identify protocol
ceph-mds 1394 root 158u sock 0,6 0t0 14350 can't
identify protocol
ceph-mds 1394 root 159u sock 0,6 0t0 15519 can't
identify protocol
ceph-mds 1394 root 160u sock 0,6 0t0 15668 can't
identify protocol
ceph-mds 1394 root 161u sock 0,6 0t0 15679 can't
identify protocol
ceph-mds 1394 root 162u sock 0,6 0t0 15727 can't
identify protocol
ceph-mds 1394 root 163u sock 0,6 0t0 15740 can't
identify protocol
ceph-mds 1394 root 164u sock 0,6 0t0 15757 can't
identify protocol
ceph-mds 1394 root 165u sock 0,6 0t0 15759 can't
identify protocol
ceph-mds 1394 root 166u sock 0,6 0t0 15761 can't
identify protocol
ceph-mds 1394 root 167u sock 0,6 0t0 15763 can't
identify protocol
ceph-mds 1394 root 168u sock 0,6 0t0 15765 can't
identify protocol
ceph-mds 1394 root 169u sock 0,6 0t0 15767 can't
identify protocol
ceph-mds 1394 root 170u sock 0,6 0t0 15769 can't
identify protocol
ceph-mds 1394 root 171u sock 0,6 0t0 15771 can't
identify protocol
ceph-mds 1394 root 172u sock 0,6 0t0 15773 can't
identify protocol
ceph-mds 1394 root 173u sock 0,6 0t0 15775 can't
identify protocol
ceph-mds 1394 root 174u sock 0,6 0t0 15777 can't
identify protocol
ceph-mds 1394 root 175u sock 0,6 0t0 15779 can't
identify protocol
ceph-mds 1394 root 176u sock 0,6 0t0 15781 can't
identify protocol
ceph-mds 1394 root 177u sock 0,6 0t0 15783 can't
identify protocol
ceph-mds 1394 root 178u sock 0,6 0t0 15785 can't
identify protocol
ceph-mds 1394 root 179u sock 0,6 0t0 15795 can't
identify protocol
ceph-mds 1394 root 180u sock 0,6 0t0 15796 can't
identify protocol
ceph-mds 1394 root 181u sock 0,6 0t0 15797 can't
identify protocol
ceph-mds 1394 root 182u sock 0,6 0t0 15798 can't
identify protocol
ceph-mds 1394 root 183u sock 0,6 0t0 15800 can't
identify protocol
ceph-mds 1394 root 184u sock 0,6 0t0 15802 can't
identify protocol
ceph-mds 1394 root 185u sock 0,6 0t0 15804 can't
identify protocol
ceph-mds 1394 root 186u sock 0,6 0t0 15806 can't
identify protocol
ceph-mds 1394 root 187u sock 0,6 0t0 15809 can't
identify protocol
ceph-mds 1394 root 188u sock 0,6 0t0 15810 can't
identify protocol
ceph-mds 1394 root 189u sock 0,6 0t0 15812 can't
identify protocol
ceph-mds 1394 root 190u sock 0,6 0t0 15814 can't
identify protocol
ceph-mds 1394 root 191u sock 0,6 0t0 15816 can't
identify protocol
ceph-mds 1394 root 192u sock 0,6 0t0 15818 can't
identify protocol
ceph-mds 1394 root 193u sock 0,6 0t0 15821 can't
identify protocol
ceph-mds 1394 root 194u sock 0,6 0t0 15822 can't
identify protocol
ceph-mds 1394 root 195u sock 0,6 0t0 15824 can't
identify protocol
ceph-mds 1394 root 196u sock 0,6 0t0 15826 can't
identify protocol
ceph-mds 1394 root 197u sock 0,6 0t0 15828 can't
identify protocol
ceph-mds 1394 root 198u sock 0,6 0t0 15830 can't
identify protocol
ceph-mds 1394 root 199u sock 0,6 0t0 15852 can't
identify protocol
ceph-mds 1394 root 200u sock 0,6 0t0 15854 can't
identify protocol
ceph-mds 1394 root 201u sock 0,6 0t0 15856 can't
identify protocol
ceph-mds 1394 root 202u sock 0,6 0t0 15858 can't
identify protocol
ceph-mds 1394 root 203u sock 0,6 0t0 15860 can't
identify protocol
ceph-mds 1394 root 204u sock 0,6 0t0 15866 can't
identify protocol
ceph-mds 1394 root 205u sock 0,6 0t0 15873 can't
identify protocol
ceph-mds 1394 root 206u sock 0,6 0t0 15875 can't
identify protocol
ceph-mds 1394 root 207u sock 0,6 0t0 15877 can't
identify protocol
ceph-mds 1394 root 208u sock 0,6 0t0 15879 can't
identify protocol
ceph-mds 1394 root 209u sock 0,6 0t0 15881 can't
identify protocol
ceph-mds 1394 root 210u sock 0,6 0t0 15895 can't
identify protocol
ceph-mds 1394 root 211u sock 0,6 0t0 15896 can't
identify protocol
ceph-mds 1394 root 212u sock 0,6 0t0 15897 can't
identify protocol
ceph-mds 1394 root 213u sock 0,6 0t0 15898 can't
identify protocol
ceph-mds 1394 root 214u sock 0,6 0t0 15899 can't
identify protocol
ceph-mds 1394 root 215u sock 0,6 0t0 15900 can't
identify protocol
ceph-mds 1394 root 216u sock 0,6 0t0 15901 can't
identify protocol
ceph-mds 1394 root 217u sock 0,6 0t0 15902 can't
identify protocol
ceph-mds 1394 root 218u sock 0,6 0t0 15904 can't
identify protocol
ceph-mds 1394 root 219u sock 0,6 0t0 15906 can't
identify protocol
ceph-mds 1394 root 220u sock 0,6 0t0 15908 can't
identify protocol
ceph-mds 1394 root 221u sock 0,6 0t0 15910 can't
identify protocol
ceph-mds 1394 root 222u sock 0,6 0t0 15912 can't
identify protocol
ceph-mds 1394 root 223u sock 0,6 0t0 15914 can't
identify protocol
ceph-mds 1394 root 224u sock 0,6 0t0 15926 can't
identify protocol
ceph-mds 1394 root 225u sock 0,6 0t0 15928 can't
identify protocol
ceph-mds 1394 root 226u sock 0,6 0t0 15930 can't
identify protocol
ceph-mds 1394 root 227u sock 0,6 0t0 15932 can't
identify protocol
ceph-mds 1394 root 228u sock 0,6 0t0 15934 can't
identify protocol
ceph-mds 1394 root 229u sock 0,6 0t0 15936 can't
identify protocol
ceph-mds 1394 root 230u sock 0,6 0t0 15943 can't
identify protocol
ceph-mds 1394 root 231u sock 0,6 0t0 15945 can't
identify protocol
ceph-mds 1394 root 232u sock 0,6 0t0 15947 can't
identify protocol
ceph-mds 1394 root 233u sock 0,6 0t0 15960 can't
identify protocol
ceph-mds 1394 root 234u sock 0,6 0t0 15961 can't
identify protocol
ceph-mds 1394 root 235u sock 0,6 0t0 15962 can't
identify protocol
ceph-mds 1394 root 236u sock 0,6 0t0 15963 can't
identify protocol
ceph-mds 1394 root 237u sock 0,6 0t0 15964 can't
identify protocol
ceph-mds 1394 root 238u sock 0,6 0t0 15965 can't
identify protocol
ceph-mds 1394 root 239u sock 0,6 0t0 15966 can't
identify protocol
ceph-mds 1394 root 240u sock 0,6 0t0 15968 can't
identify protocol
ceph-mds 1394 root 241u sock 0,6 0t0 15970 can't
identify protocol
ceph-mds 1394 root 242u sock 0,6 0t0 15973 can't
identify protocol
ceph-mds 1394 root 243u sock 0,6 0t0 15974 can't
identify protocol
ceph-mds 1394 root 244u sock 0,6 0t0 15976 can't
identify protocol
ceph-mds 1394 root 245u sock 0,6 0t0 15984 can't
identify protocol
ceph-mds 1394 root 246u sock 0,6 0t0 15985 can't
identify protocol
ceph-mds 1394 root 247u sock 0,6 0t0 16002 can't
identify protocol
ceph-mds 1394 root 248u sock 0,6 0t0 16004 can't
identify protocol
ceph-mds 1394 root 249u sock 0,6 0t0 16006 can't
identify protocol
ceph-mds 1394 root 250u sock 0,6 0t0 16008 can't
identify protocol
ceph-mds 1394 root 251u sock 0,6 0t0 16015 can't
identify protocol
ceph-mds 1394 root 252u sock 0,6 0t0 16017 can't
identify protocol
ceph-mds 1394 root 253u sock 0,6 0t0 16019 can't
identify protocol
ceph-mds 1394 root 254u sock 0,6 0t0 16021 can't
identify protocol
ceph-mds 1394 root 255u sock 0,6 0t0 16023 can't
identify protocol
ceph-mds 1394 root 256u sock 0,6 0t0 16035 can't
identify protocol
ceph-mds 1394 root 257u sock 0,6 0t0 16036 can't
identify protocol
ceph-mds 1394 root 258u sock 0,6 0t0 16037 can't
identify protocol
ceph-mds 1394 root 259u sock 0,6 0t0 16038 can't
identify protocol
ceph-mds 1394 root 260u sock 0,6 0t0 16039 can't
identify protocol
ceph-mds 1394 root 261u sock 0,6 0t0 16040 can't
identify protocol
ceph-mds 1394 root 262u sock 0,6 0t0 16042 can't
identify protocol
ceph-mds 1394 root 263u sock 0,6 0t0 16044 can't
identify protocol
ceph-mds 1394 root 264u sock 0,6 0t0 16046 can't
identify protocol
ceph-mds 1394 root 265u sock 0,6 0t0 16048 can't
identify protocol
ceph-mds 1394 root 266u sock 0,6 0t0 16050 can't
identify protocol
ceph-mds 1394 root 267u sock 0,6 0t0 16057 can't
identify protocol
ceph-mds 1394 root 268u sock 0,6 0t0 16081 can't
identify protocol
ceph-mds 1394 root 269u sock 0,6 0t0 16083 can't
identify protocol
ceph-mds 1394 root 270u sock 0,6 0t0 16095 can't
identify protocol
ceph-mds 1394 root 271u sock 0,6 0t0 16097 can't
identify protocol
ceph-mds 1394 root 272u sock 0,6 0t0 16099 can't
identify protocol
ceph-mds 1394 root 273u sock 0,6 0t0 16106 can't
identify protocol
ceph-mds 1394 root 274u sock 0,6 0t0 16108 can't
identify protocol
ceph-mds 1394 root 275u sock 0,6 0t0 16115 can't
identify protocol
ceph-mds 1394 root 276u sock 0,6 0t0 16117 can't
identify protocol
ceph-mds 1394 root 277u sock 0,6 0t0 16120 can't
identify protocol
ceph-mds 1394 root 278u sock 0,6 0t0 16121 can't
identify protocol
ceph-mds 1394 root 279u sock 0,6 0t0 16123 can't
identify protocol
ceph-mds 1394 root 280u sock 0,6 0t0 16125 can't
identify protocol
ceph-mds 1394 root 281u sock 0,6 0t0 16128 can't
identify protocol
ceph-mds 1394 root 282u sock 0,6 0t0 16130 can't
identify protocol
ceph-mds 1394 root 283u sock 0,6 0t0 16131 can't
identify protocol
ceph-mds 1394 root 284u sock 0,6 0t0 16133 can't
identify protocol
ceph-mds 1394 root 285u sock 0,6 0t0 16135 can't
identify protocol
ceph-mds 1394 root 286u sock 0,6 0t0 16137 can't
identify protocol
ceph-mds 1394 root 287u sock 0,6 0t0 16149 can't
identify protocol
ceph-mds 1394 root 288u sock 0,6 0t0 16151 can't
identify protocol
ceph-mds 1394 root 289u sock 0,6 0t0 16153 can't
identify protocol
ceph-mds 1394 root 290u sock 0,6 0t0 16161 can't
identify protocol
ceph-mds 1394 root 291u sock 0,6 0t0 16162 can't
identify protocol
ceph-mds 1394 root 292u sock 0,6 0t0 16174 can't
identify protocol
ceph-mds 1394 root 293u sock 0,6 0t0 16176 can't
identify protocol
ceph-mds 1394 root 294u sock 0,6 0t0 16183 can't
identify protocol
ceph-mds 1394 root 295u sock 0,6 0t0 16185 can't
identify protocol
ceph-mds 1394 root 296u sock 0,6 0t0 16187 can't
identify protocol
ceph-mds 1394 root 297u sock 0,6 0t0 16196 can't
identify protocol
ceph-mds 1394 root 298u sock 0,6 0t0 16197 can't
identify protocol
ceph-mds 1394 root 299u sock 0,6 0t0 16199 can't
identify protocol
ceph-mds 1394 root 300u sock 0,6 0t0 16200 can't
identify protocol
ceph-mds 1394 root 301u sock 0,6 0t0 16202 can't
identify protocol
ceph-mds 1394 root 302u sock 0,6 0t0 16204 can't
identify protocol
ceph-mds 1394 root 303u sock 0,6 0t0 16206 can't
identify protocol
ceph-mds 1394 root 304u sock 0,6 0t0 16213 can't
identify protocol
ceph-mds 1394 root 305u sock 0,6 0t0 16214 can't
identify protocol
ceph-mds 1394 root 306u sock 0,6 0t0 16216 can't
identify protocol
ceph-mds 1394 root 307u sock 0,6 0t0 16223 can't
identify protocol
ceph-mds 1394 root 308u sock 0,6 0t0 16225 can't
identify protocol
ceph-mds 1394 root 309u sock 0,6 0t0 16227 can't
identify protocol
ceph-mds 1394 root 310u sock 0,6 0t0 16229 can't
identify protocol
ceph-mds 1394 root 311u sock 0,6 0t0 16231 can't
identify protocol
ceph-mds 1394 root 312u sock 0,6 0t0 16239 can't
identify protocol
ceph-mds 1394 root 313u sock 0,6 0t0 16240 can't
identify protocol
ceph-mds 1394 root 314u sock 0,6 0t0 16252 can't
identify protocol
ceph-mds 1394 root 315u sock 0,6 0t0 16265 can't
identify protocol
ceph-mds 1394 root 316u sock 0,6 0t0 16266 can't
identify protocol
ceph-mds 1394 root 317u sock 0,6 0t0 16273 can't
identify protocol
ceph-mds 1394 root 318u sock 0,6 0t0 16275 can't
identify protocol
ceph-mds 1394 root 319u sock 0,6 0t0 16282 can't
identify protocol
ceph-mds 1394 root 320u sock 0,6 0t0 16284 can't
identify protocol
ceph-mds 1394 root 321u sock 0,6 0t0 16286 can't
identify protocol
ceph-mds 1394 root 322u sock 0,6 0t0 16288 can't
identify protocol
ceph-mds 1394 root 323u sock 0,6 0t0 16290 can't
identify protocol
ceph-mds 1394 root 324u sock 0,6 0t0 16297 can't
identify protocol
ceph-mds 1394 root 325u sock 0,6 0t0 16299 can't
identify protocol
ceph-mds 1394 root 326u sock 0,6 0t0 16301 can't
identify protocol
ceph-mds 1394 root 327u sock 0,6 0t0 16303 can't
identify protocol
ceph-mds 1394 root 328u sock 0,6 0t0 16305 can't
identify protocol
ceph-mds 1394 root 329u sock 0,6 0t0 16307 can't
identify protocol
ceph-mds 1394 root 330u sock 0,6 0t0 16310 can't
identify protocol
ceph-mds 1394 root 331u sock 0,6 0t0 16311 can't
identify protocol
ceph-mds 1394 root 332u sock 0,6 0t0 16313 can't
identify protocol
ceph-mds 1394 root 333u sock 0,6 0t0 16315 can't
identify protocol
ceph-mds 1394 root 334u sock 0,6 0t0 16317 can't
identify protocol
ceph-mds 1394 root 335u sock 0,6 0t0 18794 can't
identify protocol
ceph-mds 1394 root 336u sock 0,6 0t0 16346 can't
identify protocol
ceph-mds 1394 root 337u sock 0,6 0t0 16358 can't
identify protocol
ceph-mds 1394 root 338u sock 0,6 0t0 16360 can't
identify protocol
ceph-mds 1394 root 339u sock 0,6 0t0 16362 can't
identify protocol
ceph-mds 1394 root 340u sock 0,6 0t0 16370 can't
identify protocol
ceph-mds 1394 root 341u sock 0,6 0t0 16377 can't
identify protocol
ceph-mds 1394 root 342u sock 0,6 0t0 19386 can't
identify protocol
ceph-mds 1394 root 343u sock 0,6 0t0 19398 can't
identify protocol
ceph-mds 1394 root 344u sock 0,6 0t0 19435 can't
identify protocol
ceph-mds 1394 root 345u sock 0,6 0t0 19441 can't
identify protocol
ceph-mds 1394 root 346u sock 0,6 0t0 19489 can't
identify protocol
ceph-mds 1394 root 347u sock 0,6 0t0 19502 can't
identify protocol
ceph-mds 1394 root 348u sock 0,6 0t0 19519 can't
identify protocol
ceph-mds 1394 root 349u sock 0,6 0t0 19521 can't
identify protocol
ceph-mds 1394 root 350u sock 0,6 0t0 19523 can't
identify protocol
ceph-mds 1394 root 351u sock 0,6 0t0 19525 can't
identify protocol
ceph-mds 1394 root 352u sock 0,6 0t0 19527 can't
identify protocol
ceph-mds 1394 root 353u sock 0,6 0t0 19529 can't
identify protocol
ceph-mds 1394 root 354u sock 0,6 0t0 19531 can't
identify protocol
ceph-mds 1394 root 355u sock 0,6 0t0 19533 can't
identify protocol
ceph-mds 1394 root 356u sock 0,6 0t0 19535 can't
identify protocol
ceph-mds 1394 root 357u sock 0,6 0t0 19537 can't
identify protocol
ceph-mds 1394 root 358u sock 0,6 0t0 19539 can't
identify protocol
ceph-mds 1394 root 359u sock 0,6 0t0 19541 can't
identify protocol
ceph-mds 1394 root 360u sock 0,6 0t0 19543 can't
identify protocol
ceph-mds 1394 root 361u sock 0,6 0t0 19545 can't
identify protocol
ceph-mds 1394 root 362u sock 0,6 0t0 19547 can't
identify protocol
ceph-mds 1394 root 363u sock 0,6 0t0 19549 can't
identify protocol
ceph-mds 1394 root 364u sock 0,6 0t0 19551 can't
identify protocol
ceph-mds 1394 root 365u sock 0,6 0t0 19554 can't
identify protocol
ceph-mds 1394 root 366u sock 0,6 0t0 19555 can't
identify protocol
ceph-mds 1394 root 367u sock 0,6 0t0 19557 can't
identify protocol
ceph-mds 1394 root 368u sock 0,6 0t0 19559 can't
identify protocol
ceph-mds 1394 root 369u sock 0,6 0t0 19567 can't
identify protocol
ceph-mds 1394 root 370u sock 0,6 0t0 19568 can't
identify protocol
ceph-mds 1394 root 371u sock 0,6 0t0 19571 can't
identify protocol
ceph-mds 1394 root 372u sock 0,6 0t0 19572 can't
identify protocol
ceph-mds 1394 root 373u sock 0,6 0t0 19575 can't
identify protocol
ceph-mds 1394 root 374u sock 0,6 0t0 19576 can't
identify protocol
ceph-mds 1394 root 375u sock 0,6 0t0 19578 can't
identify protocol
ceph-mds 1394 root 376u sock 0,6 0t0 19580 can't
identify protocol
ceph-mds 1394 root 377u sock 0,6 0t0 19583 can't
identify protocol
ceph-mds 1394 root 378u sock 0,6 0t0 19584 can't
identify protocol
ceph-mds 1394 root 379u sock 0,6 0t0 19586 can't
identify protocol
ceph-mds 1394 root 380u sock 0,6 0t0 19588 can't
identify protocol
ceph-mds 1394 root 381u sock 0,6 0t0 19590 can't
identify protocol
ceph-mds 1394 root 382u sock 0,6 0t0 19592 can't
identify protocol
ceph-mds 1394 root 383u sock 0,6 0t0 19605 can't
identify protocol
ceph-mds 1394 root 384u sock 0,6 0t0 19607 can't
identify protocol
ceph-mds 1394 root 385u sock 0,6 0t0 19609 can't
identify protocol
ceph-mds 1394 root 386u sock 0,6 0t0 19611 can't
identify protocol
ceph-mds 1394 root 387u sock 0,6 0t0 19618 can't
identify protocol
ceph-mds 1394 root 388u sock 0,6 0t0 19620 can't
identify protocol
ceph-mds 1394 root 389u sock 0,6 0t0 19622 can't
identify protocol
ceph-mds 1394 root 390u sock 0,6 0t0 19624 can't
identify protocol
ceph-mds 1394 root 391u sock 0,6 0t0 19626 can't
identify protocol
ceph-mds 1394 root 392u sock 0,6 0t0 19628 can't
identify protocol
ceph-mds 1394 root 393u sock 0,6 0t0 19630 can't
identify protocol
ceph-mds 1394 root 394u sock 0,6 0t0 19632 can't
identify protocol
ceph-mds 1394 root 395u sock 0,6 0t0 19641 can't
identify protocol
ceph-mds 1394 root 396u sock 0,6 0t0 19642 can't
identify protocol
ceph-mds 1394 root 397u sock 0,6 0t0 19643 can't
identify protocol
ceph-mds 1394 root 398u sock 0,6 0t0 19645 can't
identify protocol
ceph-mds 1394 root 399u sock 0,6 0t0 19647 can't
identify protocol
ceph-mds 1394 root 400u sock 0,6 0t0 19649 can't
identify protocol
ceph-mds 1394 root 401u sock 0,6 0t0 19651 can't
identify protocol
ceph-mds 1394 root 402u sock 0,6 0t0 19653 can't
identify protocol
ceph-mds 1394 root 403u sock 0,6 0t0 19666 can't
identify protocol
ceph-mds 1394 root 404u sock 0,6 0t0 19667 can't
identify protocol
ceph-mds 1394 root 405u sock 0,6 0t0 19668 can't
identify protocol
ceph-mds 1394 root 406u sock 0,6 0t0 19675 can't
identify protocol
ceph-mds 1394 root 407u sock 0,6 0t0 19677 can't
identify protocol
ceph-mds 1394 root 408u sock 0,6 0t0 19679 can't
identify protocol
ceph-mds 1394 root 409u sock 0,6 0t0 19681 can't
identify protocol
ceph-mds 1394 root 410u sock 0,6 0t0 19683 can't
identify protocol
ceph-mds 1394 root 411u sock 0,6 0t0 19685 can't
identify protocol
ceph-mds 1394 root 412u sock 0,6 0t0 19687 can't
identify protocol
ceph-mds 1394 root 413u sock 0,6 0t0 19689 can't
identify protocol
ceph-mds 1394 root 414u sock 0,6 0t0 19691 can't
identify protocol
ceph-mds 1394 root 415u sock 0,6 0t0 19693 can't
identify protocol
ceph-mds 1394 root 416u sock 0,6 0t0 19695 can't
identify protocol
ceph-mds 1394 root 417u sock 0,6 0t0 19697 can't
identify protocol
ceph-mds 1394 root 418u sock 0,6 0t0 19699 can't
identify protocol
ceph-mds 1394 root 419u sock 0,6 0t0 19701 can't
identify protocol
ceph-mds 1394 root 420u sock 0,6 0t0 19704 can't
identify protocol
ceph-mds 1394 root 421u sock 0,6 0t0 19705 can't
identify protocol
ceph-mds 1394 root 422u sock 0,6 0t0 19716 can't
identify protocol
ceph-mds 1394 root 423u sock 0,6 0t0 19718 can't
identify protocol
ceph-mds 1394 root 424u sock 0,6 0t0 19707 can't
identify protocol
ceph-mds 1394 root 425u sock 0,6 0t0 19709 can't
identify protocol
ceph-mds 1394 root 426u sock 0,6 0t0 19720 can't
identify protocol
ceph-mds 1394 root 427u sock 0,6 0t0 19727 can't
identify protocol
ceph-mds 1394 root 428u sock 0,6 0t0 19735 can't
identify protocol
ceph-mds 1394 root 429u sock 0,6 0t0 19736 can't
identify protocol
ceph-mds 1394 root 430u sock 0,6 0t0 19738 can't
identify protocol
ceph-mds 1394 root 431u sock 0,6 0t0 19740 can't
identify protocol
ceph-mds 1394 root 432u sock 0,6 0t0 19742 can't
identify protocol
ceph-mds 1394 root 433u sock 0,6 0t0 19744 can't
identify protocol
ceph-mds 1394 root 434u sock 0,6 0t0 19746 can't
identify protocol
ceph-mds 1394 root 435u sock 0,6 0t0 19748 can't
identify protocol
ceph-mds 1394 root 436u sock 0,6 0t0 19750 can't
identify protocol
ceph-mds 1394 root 437u sock 0,6 0t0 19752 can't
identify protocol
ceph-mds 1394 root 438u sock 0,6 0t0 19754 can't
identify protocol
ceph-mds 1394 root 439u sock 0,6 0t0 19756 can't
identify protocol
ceph-mds 1394 root 440u sock 0,6 0t0 19765 can't
identify protocol
ceph-mds 1394 root 441u sock 0,6 0t0 19766 can't
identify protocol
ceph-mds 1394 root 442u sock 0,6 0t0 19767 can't
identify protocol
ceph-mds 1394 root 443u sock 0,6 0t0 19769 can't
identify protocol
ceph-mds 1394 root 444u sock 0,6 0t0 19771 can't
identify protocol
ceph-mds 1394 root 445u sock 0,6 0t0 19773 can't
identify protocol
ceph-mds 1394 root 446u sock 0,6 0t0 19775 can't
identify protocol
ceph-mds 1394 root 447u sock 0,6 0t0 19777 can't
identify protocol
ceph-mds 1394 root 448u sock 0,6 0t0 19784 can't
identify protocol
ceph-mds 1394 root 449u sock 0,6 0t0 19786 can't
identify protocol
ceph-mds 1394 root 450u sock 0,6 0t0 19788 can't
identify protocol
ceph-mds 1394 root 451u sock 0,6 0t0 19795 can't
identify protocol
ceph-mds 1394 root 452u sock 0,6 0t0 19797 can't
identify protocol
ceph-mds 1394 root 453u sock 0,6 0t0 19799 can't
identify protocol
ceph-mds 1394 root 454u sock 0,6 0t0 19801 can't
identify protocol
ceph-mds 1394 root 455u sock 0,6 0t0 19803 can't
identify protocol
ceph-mds 1394 root 456u sock 0,6 0t0 19811 can't
identify protocol
ceph-mds 1394 root 457u sock 0,6 0t0 19812 can't
identify protocol
ceph-mds 1394 root 458u sock 0,6 0t0 19814 can't
identify protocol
ceph-mds 1394 root 459u sock 0,6 0t0 19816 can't
identify protocol
ceph-mds 1394 root 460u sock 0,6 0t0 19819 can't
identify protocol
ceph-mds 1394 root 461u sock 0,6 0t0 19820 can't
identify protocol
ceph-mds 1394 root 462u sock 0,6 0t0 19830 can't
identify protocol
ceph-mds 1394 root 463u sock 0,6 0t0 19831 can't
identify protocol
ceph-mds 1394 root 464u sock 0,6 0t0 19832 can't
identify protocol
ceph-mds 1394 root 465u sock 0,6 0t0 19833 can't
identify protocol
ceph-mds 1394 root 466u sock 0,6 0t0 19835 can't
identify protocol
ceph-mds 1394 root 467u sock 0,6 0t0 19837 can't
identify protocol
ceph-mds 1394 root 468u sock 0,6 0t0 19848 can't
identify protocol
ceph-mds 1394 root 469u sock 0,6 0t0 19849 can't
identify protocol
ceph-mds 1394 root 470u sock 0,6 0t0 19850 can't
identify protocol
ceph-mds 1394 root 471u sock 0,6 0t0 19851 can't
identify protocol
ceph-mds 1394 root 472u sock 0,6 0t0 19852 can't
identify protocol
ceph-mds 1394 root 473u sock 0,6 0t0 19854 can't
identify protocol
ceph-mds 1394 root 474u sock 0,6 0t0 19856 can't
identify protocol
ceph-mds 1394 root 475u sock 0,6 0t0 19863 can't
identify protocol
ceph-mds 1394 root 476u sock 0,6 0t0 19865 can't
identify protocol
ceph-mds 1394 root 477u sock 0,6 0t0 19867 can't
identify protocol
ceph-mds 1394 root 478u sock 0,6 0t0 19869 can't
identify protocol
ceph-mds 1394 root 479u sock 0,6 0t0 19871 can't
identify protocol
ceph-mds 1394 root 480u sock 0,6 0t0 19878 can't
identify protocol
ceph-mds 1394 root 481u sock 0,6 0t0 19880 can't
identify protocol
ceph-mds 1394 root 482u sock 0,6 0t0 19882 can't
identify protocol
ceph-mds 1394 root 483u sock 0,6 0t0 19884 can't
identify protocol
ceph-mds 1394 root 484u sock 0,6 0t0 19886 can't
identify protocol
ceph-mds 1394 root 485u sock 0,6 0t0 19888 can't
identify protocol
ceph-mds 1394 root 486u sock 0,6 0t0 19890 can't
identify protocol
ceph-mds 1394 root 487u sock 0,6 0t0 19892 can't
identify protocol
ceph-mds 1394 root 488u sock 0,6 0t0 19894 can't
identify protocol
ceph-mds 1394 root 489u sock 0,6 0t0 19902 can't
identify protocol
ceph-mds 1394 root 490u sock 0,6 0t0 19903 can't
identify protocol
ceph-mds 1394 root 491u sock 0,6 0t0 19905 can't
identify protocol
ceph-mds 1394 root 492u sock 0,6 0t0 19907 can't
identify protocol
ceph-mds 1394 root 493u sock 0,6 0t0 19909 can't
identify protocol
ceph-mds 1394 root 494u sock 0,6 0t0 19911 can't
identify protocol
ceph-mds 1394 root 495u sock 0,6 0t0 19913 can't
identify protocol
ceph-mds 1394 root 496u sock 0,6 0t0 19915 can't
identify protocol
ceph-mds 1394 root 497u sock 0,6 0t0 19922 can't
identify protocol
ceph-mds 1394 root 498u sock 0,6 0t0 19924 can't
identify protocol
ceph-mds 1394 root 499u sock 0,6 0t0 19926 can't
identify protocol
ceph-mds 1394 root 500u sock 0,6 0t0 19934 can't
identify protocol
ceph-mds 1394 root 501u sock 0,6 0t0 19935 can't
identify protocol
ceph-mds 1394 root 502u sock 0,6 0t0 19937 can't
identify protocol
ceph-mds 1394 root 503u sock 0,6 0t0 19943 can't
identify protocol
ceph-mds 1394 root 504u sock 0,6 0t0 19951 can't
identify protocol
ceph-mds 1394 root 505u sock 0,6 0t0 19952 can't
identify protocol
ceph-mds 1394 root 506u sock 0,6 0t0 19954 can't
identify protocol
ceph-mds 1394 root 507u sock 0,6 0t0 19956 can't
identify protocol
ceph-mds 1394 root 508u sock 0,6 0t0 19958 can't
identify protocol
ceph-mds 1394 root 509u sock 0,6 0t0 19960 can't
identify protocol
ceph-mds 1394 root 510u sock 0,6 0t0 19962 can't
identify protocol
ceph-mds 1394 root 511u IPv4 41250 0t0 TCP
192.168.141.123:34406->192.168.141.129:6800 (ESTABLISHED)
ceph-mds 1394 root 512u sock 0,6 0t0 19974 can't
identify protocol
ceph-mds 1394 root 513u sock 0,6 0t0 19975 can't
identify protocol
ceph-mds 1394 root 514u sock 0,6 0t0 19976 can't
identify protocol
ceph-mds 1394 root 515u sock 0,6 0t0 19977 can't
identify protocol
ceph-mds 1394 root 516u sock 0,6 0t0 20409 can't
identify protocol
ceph-mds 1394 root 517u sock 0,6 0t0 19986 can't
identify protocol
ceph-mds 1394 root 518u sock 0,6 0t0 19988 can't
identify protocol
ceph-mds 1394 root 519u sock 0,6 0t0 20446 can't
identify protocol
ceph-mds 1394 root 520u sock 0,6 0t0 19997 can't
identify protocol
ceph-mds 1394 root 521u sock 0,6 0t0 19999 can't
identify protocol
ceph-mds 1394 root 522u sock 0,6 0t0 20001 can't
identify protocol
ceph-mds 1394 root 523u sock 0,6 0t0 20003 can't
identify protocol
ceph-mds 1394 root 524u sock 0,6 0t0 20010 can't
identify protocol
ceph-mds 1394 root 525u sock 0,6 0t0 20012 can't
identify protocol
ceph-mds 1394 root 526u sock 0,6 0t0 20014 can't
identify protocol
ceph-mds 1394 root 527u sock 0,6 0t0 20022 can't
identify protocol
ceph-mds 1394 root 528u sock 0,6 0t0 20023 can't
identify protocol
ceph-mds 1394 root 529u sock 0,6 0t0 20025 can't
identify protocol
ceph-mds 1394 root 530u sock 0,6 0t0 20032 can't
identify protocol
ceph-mds 1394 root 531u sock 0,6 0t0 20039 can't
identify protocol
ceph-mds 1394 root 532u sock 0,6 0t0 20046 can't
identify protocol
ceph-mds 1394 root 533u sock 0,6 0t0 20110 can't
identify protocol
ceph-mds 1394 root 534u sock 0,6 0t0 20134 can't
identify protocol
ceph-mds 1394 root 535u sock 0,6 0t0 20146 can't
identify protocol
ceph-mds 1394 root 536u sock 0,6 0t0 20158 can't
identify protocol
ceph-mds 1394 root 537u sock 0,6 0t0 20170 can't
identify protocol
ceph-mds 1394 root 538u sock 0,6 0t0 20192 can't
identify protocol
ceph-mds 1394 root 539u sock 0,6 0t0 40930 can't
identify protocol
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 19:05 ` Noah Watkins
@ 2011-12-05 19:11 ` Sage Weil
2011-12-05 19:36 ` Tommi Virtanen
1 sibling, 0 replies; 12+ messages in thread
From: Sage Weil @ 2011-12-05 19:11 UTC (permalink / raw)
To: Noah Watkins; +Cc: Tommi Virtanen, Noah Watkins, ceph-devel
Hmm, the other bit that would be useful would be a log with 'debug ms =
20' that is associated with these dumps. Then we can correlate the unused
sockets with the log and see where they came from.
Thanks!
sage
On Mon, 5 Dec 2011, Noah Watkins wrote:
> On 12/05/2011 11:03 AM, Tommi Virtanen wrote:
> > Can we get a "sudo lsof -n -p 1394" too? That would tell a little bit
> > more about what the sockets are actually for..
> >
> root@issdm-23:~# lsof -n -p 1394
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> ceph-mds 1394 root cwd DIR 8,1 4096 2 /
> ceph-mds 1394 root rtd DIR 8,1 4096 2 /
> ceph-mds 1394 root txt REG 8,1 4577640 7097096
> /usr/bin/ceph-mds
> ceph-mds 1394 root mem REG 8,1 42856 7096319
> /usr/lib/libunwind.so.7.0.0
> ceph-mds 1394 root mem REG 8,1 1572232 9178738
> /lib/libc-2.12.1.so
> ceph-mds 1394 root mem REG 8,1 88384 9175100
> /lib/libgcc_s.so.1
> ceph-mds 1394 root mem REG 8,1 534832 9178742
> /lib/libm-2.12.1.so
> ceph-mds 1394 root mem REG 8,1 986888 7082428
> /usr/lib/libstdc++.so.6.0.14
> ceph-mds 1394 root mem REG 8,1 249080 7096323
> /usr/lib/libtcmalloc.so.0.0.0
> ceph-mds 1394 root mem REG 8,1 31744 9178754
> /lib/librt-2.12.1.so
> ceph-mds 1394 root mem REG 8,1 19008 9175163
> /lib/libuuid.so.1.3.0
> ceph-mds 1394 root mem REG 8,1 4946936 7096152
> /usr/lib/libcrypto++.so.8.0.0
> ceph-mds 1394 root mem REG 8,1 136067 9178752
> /lib/libpthread-2.12.1.so
> ceph-mds 1394 root mem REG 8,1 141072 9178735
> /lib/ld-2.12.1.so
> ceph-mds 1394 root 0r CHR 1,3 0t0 4372 /dev/null
> ceph-mds 1394 root 1r CHR 1,3 0t0 4372 /dev/null
> ceph-mds 1394 root 2r CHR 1,3 0t0 4372 /dev/null
> ceph-mds 1394 root 3w REG 8,1 14761050 6034073
> /var/log/ceph/mds.a.log
> ceph-mds 1394 root 4u IPv4 7837 0t0 TCP *:6800
> (LISTEN)
> ceph-mds 1394 root 5r FIFO 0,8 0t0 7841 pipe
> ceph-mds 1394 root 6w FIFO 0,8 0t0 7841 pipe
> ceph-mds 1394 root 7u unix 0xffff88021a623a80 0t0 7842
> /var/run/ceph/mds.a.asok
> ceph-mds 1394 root 8u IPv4 7850 0t0 TCP
> 192.168.141.123:38046->192.168.141.123:6789 (ESTABLISHED)
> ceph-mds 1394 root 9u IPv4 41245 0t0 TCP
> 192.168.141.123:58242->192.168.141.135:6800 (ESTABLISHED)
> ceph-mds 1394 root 10u IPv4 41249 0t0 TCP
> 192.168.141.123:39475->192.168.141.128:6800 (ESTABLISHED)
> ceph-mds 1394 root 11u IPv4 40004 0t0 TCP
> 192.168.141.123:53885->192.168.141.124:6800 (ESTABLISHED)
> ceph-mds 1394 root 12u sock 0,6 0t0 11560 can't
> identify protocol
> ceph-mds 1394 root 13u IPv4 41246 0t0 TCP
> 192.168.141.123:51006->192.168.141.132:6800 (ESTABLISHED)
> ceph-mds 1394 root 14u IPv4 40005 0t0 TCP
> 192.168.141.123:34639->192.168.141.138:6800 (ESTABLISHED)
> ceph-mds 1394 root 15u IPv4 41247 0t0 TCP
> 192.168.141.123:43558->192.168.141.125:6800 (ESTABLISHED)
> ceph-mds 1394 root 16u IPv4 41241 0t0 TCP
> 192.168.141.123:52571->192.168.141.127:6800 (ESTABLISHED)
> ceph-mds 1394 root 17u sock 0,6 0t0 12488 can't
> identify protocol
> ceph-mds 1394 root 18u sock 0,6 0t0 10214 can't
> identify protocol
> ceph-mds 1394 root 19u IPv4 10656 0t0 TCP
> 192.168.141.123:6800->192.168.141.123:46078 (ESTABLISHED)
> ceph-mds 1394 root 20u IPv4 10847 0t0 TCP
> 192.168.141.123:6800->192.168.141.130:37586 (ESTABLISHED)
> ceph-mds 1394 root 21u IPv4 10849 0t0 TCP
> 192.168.141.123:6800->192.168.141.135:44058 (ESTABLISHED)
> ceph-mds 1394 root 22u IPv4 10856 0t0 TCP
> 192.168.141.123:6800->192.168.141.129:34711 (ESTABLISHED)
> ceph-mds 1394 root 23u IPv4 10858 0t0 TCP
> 192.168.141.123:6800->192.168.141.137:35032 (ESTABLISHED)
> ceph-mds 1394 root 24u IPv4 10863 0t0 TCP
> 192.168.141.123:6800->192.168.141.131:34520 (ESTABLISHED)
> ceph-mds 1394 root 25u IPv4 10872 0t0 TCP
> 192.168.141.123:6800->192.168.141.122:44304 (ESTABLISHED)
> ceph-mds 1394 root 26u IPv4 10873 0t0 TCP
> 192.168.141.123:6800->192.168.141.128:51902 (ESTABLISHED)
> ceph-mds 1394 root 27u IPv4 10874 0t0 TCP
> 192.168.141.123:6800->192.168.141.125:60607 (ESTABLISHED)
> ceph-mds 1394 root 28u IPv4 10876 0t0 TCP
> 192.168.141.123:6800->192.168.141.124:39850 (ESTABLISHED)
> ceph-mds 1394 root 29u IPv4 10878 0t0 TCP
> 192.168.141.123:6800->192.168.141.127:43072 (ESTABLISHED)
> ceph-mds 1394 root 30u IPv4 10892 0t0 TCP
> 192.168.141.123:6800->192.168.141.132:41561 (ESTABLISHED)
> ceph-mds 1394 root 31u IPv4 10919 0t0 TCP
> 192.168.141.123:6800->192.168.141.138:36645 (ESTABLISHED)
> ceph-mds 1394 root 32u IPv4 41242 0t0 TCP
> 192.168.141.123:56058->192.168.141.137:6800 (ESTABLISHED)
> ceph-mds 1394 root 33u sock 0,6 0t0 11572 can't
> identify protocol
> ceph-mds 1394 root 34u sock 0,6 0t0 12540 can't
> identify protocol
> ceph-mds 1394 root 35u sock 0,6 0t0 20392 can't
> identify protocol
> ceph-mds 1394 root 36u IPv4 41244 0t0 TCP
> 192.168.141.123:47859->192.168.141.122:6800 (ESTABLISHED)
> ceph-mds 1394 root 37u sock 0,6 0t0 11612 can't
> identify protocol
> ceph-mds 1394 root 38u sock 0,6 0t0 11629 can't
> identify protocol
> ceph-mds 1394 root 39u sock 0,6 0t0 11689 can't
> identify protocol
> ceph-mds 1394 root 40u sock 0,6 0t0 11716 can't
> identify protocol
> ceph-mds 1394 root 41u sock 0,6 0t0 11727 can't
> identify protocol
> ceph-mds 1394 root 42u sock 0,6 0t0 11748 can't
> identify protocol
> ceph-mds 1394 root 43u sock 0,6 0t0 11761 can't
> identify protocol
> ceph-mds 1394 root 44u sock 0,6 0t0 11778 can't
> identify protocol
> ceph-mds 1394 root 45u sock 0,6 0t0 11780 can't
> identify protocol
> ceph-mds 1394 root 46u sock 0,6 0t0 11787 can't
> identify protocol
> ceph-mds 1394 root 47u IPv4 41248 0t0 TCP
> 192.168.141.123:54637->192.168.141.131:6800 (ESTABLISHED)
> ceph-mds 1394 root 48u IPv4 41243 0t0 TCP
> 192.168.141.123:48293->192.168.141.130:6800 (ESTABLISHED)
> ceph-mds 1394 root 49u sock 0,6 0t0 12551 can't
> identify protocol
> ceph-mds 1394 root 50u sock 0,6 0t0 12605 can't
> identify protocol
> ceph-mds 1394 root 51u sock 0,6 0t0 12618 can't
> identify protocol
> ceph-mds 1394 root 52u sock 0,6 0t0 12630 can't
> identify protocol
> ceph-mds 1394 root 53u sock 0,6 0t0 12632 can't
> identify protocol
> ceph-mds 1394 root 54u sock 0,6 0t0 12634 can't
> identify protocol
> ceph-mds 1394 root 55u sock 0,6 0t0 12636 can't
> identify protocol
> ceph-mds 1394 root 56u sock 0,6 0t0 12639 can't
> identify protocol
> ceph-mds 1394 root 57u sock 0,6 0t0 12640 can't
> identify protocol
> ceph-mds 1394 root 58u sock 0,6 0t0 12642 can't
> identify protocol
> ceph-mds 1394 root 59u sock 0,6 0t0 12644 can't
> identify protocol
> ceph-mds 1394 root 60u sock 0,6 0t0 12646 can't
> identify protocol
> ceph-mds 1394 root 61u sock 0,6 0t0 12648 can't
> identify protocol
> ceph-mds 1394 root 62u sock 0,6 0t0 12651 can't
> identify protocol
> ceph-mds 1394 root 63u sock 0,6 0t0 12652 can't
> identify protocol
> ceph-mds 1394 root 64u sock 0,6 0t0 12654 can't
> identify protocol
> ceph-mds 1394 root 65u sock 0,6 0t0 12656 can't
> identify protocol
> ceph-mds 1394 root 66u sock 0,6 0t0 12659 can't
> identify protocol
> ceph-mds 1394 root 67u sock 0,6 0t0 12660 can't
> identify protocol
> ceph-mds 1394 root 68u sock 0,6 0t0 12662 can't
> identify protocol
> ceph-mds 1394 root 69u sock 0,6 0t0 12664 can't
> identify protocol
> ceph-mds 1394 root 70u sock 0,6 0t0 12666 can't
> identify protocol
> ceph-mds 1394 root 71u sock 0,6 0t0 12674 can't
> identify protocol
> ceph-mds 1394 root 72u sock 0,6 0t0 12675 can't
> identify protocol
> ceph-mds 1394 root 73u sock 0,6 0t0 12682 can't
> identify protocol
> ceph-mds 1394 root 74u sock 0,6 0t0 12684 can't
> identify protocol
> ceph-mds 1394 root 75u sock 0,6 0t0 12687 can't
> identify protocol
> ceph-mds 1394 root 76u sock 0,6 0t0 12688 can't
> identify protocol
> ceph-mds 1394 root 77u sock 0,6 0t0 12691 can't
> identify protocol
> ceph-mds 1394 root 78u sock 0,6 0t0 12692 can't
> identify protocol
> ceph-mds 1394 root 79u sock 0,6 0t0 12694 can't
> identify protocol
> ceph-mds 1394 root 80u sock 0,6 0t0 12696 can't
> identify protocol
> ceph-mds 1394 root 81u sock 0,6 0t0 12698 can't
> identify protocol
> ceph-mds 1394 root 82u sock 0,6 0t0 12700 can't
> identify protocol
> ceph-mds 1394 root 83u sock 0,6 0t0 12702 can't
> identify protocol
> ceph-mds 1394 root 84u sock 0,6 0t0 12704 can't
> identify protocol
> ceph-mds 1394 root 85u sock 0,6 0t0 12706 can't
> identify protocol
> ceph-mds 1394 root 86u sock 0,6 0t0 12708 can't
> identify protocol
> ceph-mds 1394 root 87u sock 0,6 0t0 12732 can't
> identify protocol
> ceph-mds 1394 root 88u sock 0,6 0t0 12734 can't
> identify protocol
> ceph-mds 1394 root 89u sock 0,6 0t0 12746 can't
> identify protocol
> ceph-mds 1394 root 90u sock 0,6 0t0 12748 can't
> identify protocol
> ceph-mds 1394 root 91u sock 0,6 0t0 12750 can't
> identify protocol
> ceph-mds 1394 root 92u sock 0,6 0t0 12752 can't
> identify protocol
> ceph-mds 1394 root 93u sock 0,6 0t0 12754 can't
> identify protocol
> ceph-mds 1394 root 94u sock 0,6 0t0 12756 can't
> identify protocol
> ceph-mds 1394 root 95u sock 0,6 0t0 12758 can't
> identify protocol
> ceph-mds 1394 root 96u sock 0,6 0t0 12760 can't
> identify protocol
> ceph-mds 1394 root 97u sock 0,6 0t0 12762 can't
> identify protocol
> ceph-mds 1394 root 98u sock 0,6 0t0 12764 can't
> identify protocol
> ceph-mds 1394 root 99u sock 0,6 0t0 12766 can't
> identify protocol
> ceph-mds 1394 root 100u sock 0,6 0t0 12769 can't
> identify protocol
> ceph-mds 1394 root 101u sock 0,6 0t0 12770 can't
> identify protocol
> ceph-mds 1394 root 102u sock 0,6 0t0 12773 can't
> identify protocol
> ceph-mds 1394 root 103u sock 0,6 0t0 12774 can't
> identify protocol
> ceph-mds 1394 root 104u sock 0,6 0t0 12776 can't
> identify protocol
> ceph-mds 1394 root 105u sock 0,6 0t0 12786 can't
> identify protocol
> ceph-mds 1394 root 106u sock 0,6 0t0 12787 can't
> identify protocol
> ceph-mds 1394 root 107u sock 0,6 0t0 12788 can't
> identify protocol
> ceph-mds 1394 root 108u sock 0,6 0t0 12789 can't
> identify protocol
> ceph-mds 1394 root 109u sock 0,6 0t0 12790 can't
> identify protocol
> ceph-mds 1394 root 110u sock 0,6 0t0 12803 can't
> identify protocol
> ceph-mds 1394 root 111u sock 0,6 0t0 12804 can't
> identify protocol
> ceph-mds 1394 root 112u sock 0,6 0t0 12816 can't
> identify protocol
> ceph-mds 1394 root 113u sock 0,6 0t0 12818 can't
> identify protocol
> ceph-mds 1394 root 114u sock 0,6 0t0 12831 can't
> identify protocol
> ceph-mds 1394 root 115u sock 0,6 0t0 12832 can't
> identify protocol
> ceph-mds 1394 root 116u sock 0,6 0t0 12834 can't
> identify protocol
> ceph-mds 1394 root 117u sock 0,6 0t0 12836 can't
> identify protocol
> ceph-mds 1394 root 118u sock 0,6 0t0 12838 can't
> identify protocol
> ceph-mds 1394 root 119u sock 0,6 0t0 12840 can't
> identify protocol
> ceph-mds 1394 root 120u sock 0,6 0t0 12842 can't
> identify protocol
> ceph-mds 1394 root 121u sock 0,6 0t0 12844 can't
> identify protocol
> ceph-mds 1394 root 122u sock 0,6 0t0 12846 can't
> identify protocol
> ceph-mds 1394 root 123u sock 0,6 0t0 12848 can't
> identify protocol
> ceph-mds 1394 root 124u sock 0,6 0t0 12851 can't
> identify protocol
> ceph-mds 1394 root 125u sock 0,6 0t0 12852 can't
> identify protocol
> ceph-mds 1394 root 126u sock 0,6 0t0 12854 can't
> identify protocol
> ceph-mds 1394 root 127u sock 0,6 0t0 12856 can't
> identify protocol
> ceph-mds 1394 root 128u sock 0,6 0t0 12858 can't
> identify protocol
> ceph-mds 1394 root 129u sock 0,6 0t0 12860 can't
> identify protocol
> ceph-mds 1394 root 130u sock 0,6 0t0 12862 can't
> identify protocol
> ceph-mds 1394 root 131u sock 0,6 0t0 12864 can't
> identify protocol
> ceph-mds 1394 root 132u sock 0,6 0t0 12866 can't
> identify protocol
> ceph-mds 1394 root 133u sock 0,6 0t0 12873 can't
> identify protocol
> ceph-mds 1394 root 134u sock 0,6 0t0 12880 can't
> identify protocol
> ceph-mds 1394 root 135u sock 0,6 0t0 12882 can't
> identify protocol
> ceph-mds 1394 root 136u sock 0,6 0t0 12894 can't
> identify protocol
> ceph-mds 1394 root 137u sock 0,6 0t0 12896 can't
> identify protocol
> ceph-mds 1394 root 138u sock 0,6 0t0 12908 can't
> identify protocol
> ceph-mds 1394 root 139u sock 0,6 0t0 12910 can't
> identify protocol
> ceph-mds 1394 root 140u sock 0,6 0t0 14170 can't
> identify protocol
> ceph-mds 1394 root 141u sock 0,6 0t0 12915 can't
> identify protocol
> ceph-mds 1394 root 142u sock 0,6 0t0 12916 can't
> identify protocol
> ceph-mds 1394 root 143u sock 0,6 0t0 12918 can't
> identify protocol
> ceph-mds 1394 root 144u sock 0,6 0t0 12920 can't
> identify protocol
> ceph-mds 1394 root 145u sock 0,6 0t0 12927 can't
> identify protocol
> ceph-mds 1394 root 146u sock 0,6 0t0 12944 can't
> identify protocol
> ceph-mds 1394 root 147u sock 0,6 0t0 12956 can't
> identify protocol
> ceph-mds 1394 root 148u sock 0,6 0t0 12958 can't
> identify protocol
> ceph-mds 1394 root 149u sock 0,6 0t0 12980 can't
> identify protocol
> ceph-mds 1394 root 150u sock 0,6 0t0 14222 can't
> identify protocol
> ceph-mds 1394 root 151u sock 0,6 0t0 14259 can't
> identify protocol
> ceph-mds 1394 root 152u sock 0,6 0t0 14302 can't
> identify protocol
> ceph-mds 1394 root 153u sock 0,6 0t0 14313 can't
> identify protocol
> ceph-mds 1394 root 154u sock 0,6 0t0 14321 can't
> identify protocol
> ceph-mds 1394 root 155u sock 0,6 0t0 14334 can't
> identify protocol
> ceph-mds 1394 root 156u sock 0,6 0t0 14346 can't
> identify protocol
> ceph-mds 1394 root 157u sock 0,6 0t0 14348 can't
> identify protocol
> ceph-mds 1394 root 158u sock 0,6 0t0 14350 can't
> identify protocol
> ceph-mds 1394 root 159u sock 0,6 0t0 15519 can't
> identify protocol
> ceph-mds 1394 root 160u sock 0,6 0t0 15668 can't
> identify protocol
> ceph-mds 1394 root 161u sock 0,6 0t0 15679 can't
> identify protocol
> ceph-mds 1394 root 162u sock 0,6 0t0 15727 can't
> identify protocol
> ceph-mds 1394 root 163u sock 0,6 0t0 15740 can't
> identify protocol
> ceph-mds 1394 root 164u sock 0,6 0t0 15757 can't
> identify protocol
> ceph-mds 1394 root 165u sock 0,6 0t0 15759 can't
> identify protocol
> ceph-mds 1394 root 166u sock 0,6 0t0 15761 can't
> identify protocol
> ceph-mds 1394 root 167u sock 0,6 0t0 15763 can't
> identify protocol
> ceph-mds 1394 root 168u sock 0,6 0t0 15765 can't
> identify protocol
> ceph-mds 1394 root 169u sock 0,6 0t0 15767 can't
> identify protocol
> ceph-mds 1394 root 170u sock 0,6 0t0 15769 can't
> identify protocol
> ceph-mds 1394 root 171u sock 0,6 0t0 15771 can't
> identify protocol
> ceph-mds 1394 root 172u sock 0,6 0t0 15773 can't
> identify protocol
> ceph-mds 1394 root 173u sock 0,6 0t0 15775 can't
> identify protocol
> ceph-mds 1394 root 174u sock 0,6 0t0 15777 can't
> identify protocol
> ceph-mds 1394 root 175u sock 0,6 0t0 15779 can't
> identify protocol
> ceph-mds 1394 root 176u sock 0,6 0t0 15781 can't
> identify protocol
> ceph-mds 1394 root 177u sock 0,6 0t0 15783 can't
> identify protocol
> ceph-mds 1394 root 178u sock 0,6 0t0 15785 can't
> identify protocol
> ceph-mds 1394 root 179u sock 0,6 0t0 15795 can't
> identify protocol
> ceph-mds 1394 root 180u sock 0,6 0t0 15796 can't
> identify protocol
> ceph-mds 1394 root 181u sock 0,6 0t0 15797 can't
> identify protocol
> ceph-mds 1394 root 182u sock 0,6 0t0 15798 can't
> identify protocol
> ceph-mds 1394 root 183u sock 0,6 0t0 15800 can't
> identify protocol
> ceph-mds 1394 root 184u sock 0,6 0t0 15802 can't
> identify protocol
> ceph-mds 1394 root 185u sock 0,6 0t0 15804 can't
> identify protocol
> ceph-mds 1394 root 186u sock 0,6 0t0 15806 can't
> identify protocol
> ceph-mds 1394 root 187u sock 0,6 0t0 15809 can't
> identify protocol
> ceph-mds 1394 root 188u sock 0,6 0t0 15810 can't
> identify protocol
> ceph-mds 1394 root 189u sock 0,6 0t0 15812 can't
> identify protocol
> ceph-mds 1394 root 190u sock 0,6 0t0 15814 can't
> identify protocol
> ceph-mds 1394 root 191u sock 0,6 0t0 15816 can't
> identify protocol
> ceph-mds 1394 root 192u sock 0,6 0t0 15818 can't
> identify protocol
> ceph-mds 1394 root 193u sock 0,6 0t0 15821 can't
> identify protocol
> ceph-mds 1394 root 194u sock 0,6 0t0 15822 can't
> identify protocol
> ceph-mds 1394 root 195u sock 0,6 0t0 15824 can't
> identify protocol
> ceph-mds 1394 root 196u sock 0,6 0t0 15826 can't
> identify protocol
> ceph-mds 1394 root 197u sock 0,6 0t0 15828 can't
> identify protocol
> ceph-mds 1394 root 198u sock 0,6 0t0 15830 can't
> identify protocol
> ceph-mds 1394 root 199u sock 0,6 0t0 15852 can't
> identify protocol
> ceph-mds 1394 root 200u sock 0,6 0t0 15854 can't
> identify protocol
> ceph-mds 1394 root 201u sock 0,6 0t0 15856 can't
> identify protocol
> ceph-mds 1394 root 202u sock 0,6 0t0 15858 can't
> identify protocol
> ceph-mds 1394 root 203u sock 0,6 0t0 15860 can't
> identify protocol
> ceph-mds 1394 root 204u sock 0,6 0t0 15866 can't
> identify protocol
> ceph-mds 1394 root 205u sock 0,6 0t0 15873 can't
> identify protocol
> ceph-mds 1394 root 206u sock 0,6 0t0 15875 can't
> identify protocol
> ceph-mds 1394 root 207u sock 0,6 0t0 15877 can't
> identify protocol
> ceph-mds 1394 root 208u sock 0,6 0t0 15879 can't
> identify protocol
> ceph-mds 1394 root 209u sock 0,6 0t0 15881 can't
> identify protocol
> ceph-mds 1394 root 210u sock 0,6 0t0 15895 can't
> identify protocol
> ceph-mds 1394 root 211u sock 0,6 0t0 15896 can't
> identify protocol
> ceph-mds 1394 root 212u sock 0,6 0t0 15897 can't
> identify protocol
> ceph-mds 1394 root 213u sock 0,6 0t0 15898 can't
> identify protocol
> ceph-mds 1394 root 214u sock 0,6 0t0 15899 can't
> identify protocol
> ceph-mds 1394 root 215u sock 0,6 0t0 15900 can't
> identify protocol
> ceph-mds 1394 root 216u sock 0,6 0t0 15901 can't
> identify protocol
> ceph-mds 1394 root 217u sock 0,6 0t0 15902 can't
> identify protocol
> ceph-mds 1394 root 218u sock 0,6 0t0 15904 can't
> identify protocol
> ceph-mds 1394 root 219u sock 0,6 0t0 15906 can't
> identify protocol
> ceph-mds 1394 root 220u sock 0,6 0t0 15908 can't
> identify protocol
> ceph-mds 1394 root 221u sock 0,6 0t0 15910 can't
> identify protocol
> ceph-mds 1394 root 222u sock 0,6 0t0 15912 can't
> identify protocol
> ceph-mds 1394 root 223u sock 0,6 0t0 15914 can't
> identify protocol
> ceph-mds 1394 root 224u sock 0,6 0t0 15926 can't
> identify protocol
> ceph-mds 1394 root 225u sock 0,6 0t0 15928 can't
> identify protocol
> ceph-mds 1394 root 226u sock 0,6 0t0 15930 can't
> identify protocol
> ceph-mds 1394 root 227u sock 0,6 0t0 15932 can't
> identify protocol
> ceph-mds 1394 root 228u sock 0,6 0t0 15934 can't
> identify protocol
> ceph-mds 1394 root 229u sock 0,6 0t0 15936 can't
> identify protocol
> ceph-mds 1394 root 230u sock 0,6 0t0 15943 can't
> identify protocol
> ceph-mds 1394 root 231u sock 0,6 0t0 15945 can't
> identify protocol
> ceph-mds 1394 root 232u sock 0,6 0t0 15947 can't
> identify protocol
> ceph-mds 1394 root 233u sock 0,6 0t0 15960 can't
> identify protocol
> ceph-mds 1394 root 234u sock 0,6 0t0 15961 can't
> identify protocol
> ceph-mds 1394 root 235u sock 0,6 0t0 15962 can't
> identify protocol
> ceph-mds 1394 root 236u sock 0,6 0t0 15963 can't
> identify protocol
> ceph-mds 1394 root 237u sock 0,6 0t0 15964 can't
> identify protocol
> ceph-mds 1394 root 238u sock 0,6 0t0 15965 can't
> identify protocol
> ceph-mds 1394 root 239u sock 0,6 0t0 15966 can't
> identify protocol
> ceph-mds 1394 root 240u sock 0,6 0t0 15968 can't
> identify protocol
> ceph-mds 1394 root 241u sock 0,6 0t0 15970 can't
> identify protocol
> ceph-mds 1394 root 242u sock 0,6 0t0 15973 can't
> identify protocol
> ceph-mds 1394 root 243u sock 0,6 0t0 15974 can't
> identify protocol
> ceph-mds 1394 root 244u sock 0,6 0t0 15976 can't
> identify protocol
> ceph-mds 1394 root 245u sock 0,6 0t0 15984 can't
> identify protocol
> ceph-mds 1394 root 246u sock 0,6 0t0 15985 can't
> identify protocol
> ceph-mds 1394 root 247u sock 0,6 0t0 16002 can't
> identify protocol
> ceph-mds 1394 root 248u sock 0,6 0t0 16004 can't
> identify protocol
> ceph-mds 1394 root 249u sock 0,6 0t0 16006 can't
> identify protocol
> ceph-mds 1394 root 250u sock 0,6 0t0 16008 can't
> identify protocol
> ceph-mds 1394 root 251u sock 0,6 0t0 16015 can't
> identify protocol
> ceph-mds 1394 root 252u sock 0,6 0t0 16017 can't
> identify protocol
> ceph-mds 1394 root 253u sock 0,6 0t0 16019 can't
> identify protocol
> ceph-mds 1394 root 254u sock 0,6 0t0 16021 can't
> identify protocol
> ceph-mds 1394 root 255u sock 0,6 0t0 16023 can't
> identify protocol
> ceph-mds 1394 root 256u sock 0,6 0t0 16035 can't
> identify protocol
> ceph-mds 1394 root 257u sock 0,6 0t0 16036 can't
> identify protocol
> ceph-mds 1394 root 258u sock 0,6 0t0 16037 can't
> identify protocol
> ceph-mds 1394 root 259u sock 0,6 0t0 16038 can't
> identify protocol
> ceph-mds 1394 root 260u sock 0,6 0t0 16039 can't
> identify protocol
> ceph-mds 1394 root 261u sock 0,6 0t0 16040 can't
> identify protocol
> ceph-mds 1394 root 262u sock 0,6 0t0 16042 can't
> identify protocol
> ceph-mds 1394 root 263u sock 0,6 0t0 16044 can't
> identify protocol
> ceph-mds 1394 root 264u sock 0,6 0t0 16046 can't
> identify protocol
> ceph-mds 1394 root 265u sock 0,6 0t0 16048 can't
> identify protocol
> ceph-mds 1394 root 266u sock 0,6 0t0 16050 can't
> identify protocol
> ceph-mds 1394 root 267u sock 0,6 0t0 16057 can't
> identify protocol
> ceph-mds 1394 root 268u sock 0,6 0t0 16081 can't
> identify protocol
> ceph-mds 1394 root 269u sock 0,6 0t0 16083 can't
> identify protocol
> ceph-mds 1394 root 270u sock 0,6 0t0 16095 can't
> identify protocol
> ceph-mds 1394 root 271u sock 0,6 0t0 16097 can't
> identify protocol
> ceph-mds 1394 root 272u sock 0,6 0t0 16099 can't
> identify protocol
> ceph-mds 1394 root 273u sock 0,6 0t0 16106 can't
> identify protocol
> ceph-mds 1394 root 274u sock 0,6 0t0 16108 can't
> identify protocol
> ceph-mds 1394 root 275u sock 0,6 0t0 16115 can't
> identify protocol
> ceph-mds 1394 root 276u sock 0,6 0t0 16117 can't
> identify protocol
> ceph-mds 1394 root 277u sock 0,6 0t0 16120 can't
> identify protocol
> ceph-mds 1394 root 278u sock 0,6 0t0 16121 can't
> identify protocol
> ceph-mds 1394 root 279u sock 0,6 0t0 16123 can't
> identify protocol
> ceph-mds 1394 root 280u sock 0,6 0t0 16125 can't
> identify protocol
> ceph-mds 1394 root 281u sock 0,6 0t0 16128 can't
> identify protocol
> ceph-mds 1394 root 282u sock 0,6 0t0 16130 can't
> identify protocol
> ceph-mds 1394 root 283u sock 0,6 0t0 16131 can't
> identify protocol
> ceph-mds 1394 root 284u sock 0,6 0t0 16133 can't
> identify protocol
> ceph-mds 1394 root 285u sock 0,6 0t0 16135 can't
> identify protocol
> ceph-mds 1394 root 286u sock 0,6 0t0 16137 can't
> identify protocol
> ceph-mds 1394 root 287u sock 0,6 0t0 16149 can't
> identify protocol
> ceph-mds 1394 root 288u sock 0,6 0t0 16151 can't
> identify protocol
> ceph-mds 1394 root 289u sock 0,6 0t0 16153 can't
> identify protocol
> ceph-mds 1394 root 290u sock 0,6 0t0 16161 can't
> identify protocol
> ceph-mds 1394 root 291u sock 0,6 0t0 16162 can't
> identify protocol
> ceph-mds 1394 root 292u sock 0,6 0t0 16174 can't
> identify protocol
> ceph-mds 1394 root 293u sock 0,6 0t0 16176 can't
> identify protocol
> ceph-mds 1394 root 294u sock 0,6 0t0 16183 can't
> identify protocol
> ceph-mds 1394 root 295u sock 0,6 0t0 16185 can't
> identify protocol
> ceph-mds 1394 root 296u sock 0,6 0t0 16187 can't
> identify protocol
> ceph-mds 1394 root 297u sock 0,6 0t0 16196 can't
> identify protocol
> ceph-mds 1394 root 298u sock 0,6 0t0 16197 can't
> identify protocol
> ceph-mds 1394 root 299u sock 0,6 0t0 16199 can't
> identify protocol
> ceph-mds 1394 root 300u sock 0,6 0t0 16200 can't
> identify protocol
> ceph-mds 1394 root 301u sock 0,6 0t0 16202 can't
> identify protocol
> ceph-mds 1394 root 302u sock 0,6 0t0 16204 can't
> identify protocol
> ceph-mds 1394 root 303u sock 0,6 0t0 16206 can't
> identify protocol
> ceph-mds 1394 root 304u sock 0,6 0t0 16213 can't
> identify protocol
> ceph-mds 1394 root 305u sock 0,6 0t0 16214 can't
> identify protocol
> ceph-mds 1394 root 306u sock 0,6 0t0 16216 can't
> identify protocol
> ceph-mds 1394 root 307u sock 0,6 0t0 16223 can't
> identify protocol
> ceph-mds 1394 root 308u sock 0,6 0t0 16225 can't
> identify protocol
> ceph-mds 1394 root 309u sock 0,6 0t0 16227 can't
> identify protocol
> ceph-mds 1394 root 310u sock 0,6 0t0 16229 can't
> identify protocol
> ceph-mds 1394 root 311u sock 0,6 0t0 16231 can't
> identify protocol
> ceph-mds 1394 root 312u sock 0,6 0t0 16239 can't
> identify protocol
> ceph-mds 1394 root 313u sock 0,6 0t0 16240 can't
> identify protocol
> ceph-mds 1394 root 314u sock 0,6 0t0 16252 can't
> identify protocol
> ceph-mds 1394 root 315u sock 0,6 0t0 16265 can't
> identify protocol
> ceph-mds 1394 root 316u sock 0,6 0t0 16266 can't
> identify protocol
> ceph-mds 1394 root 317u sock 0,6 0t0 16273 can't
> identify protocol
> ceph-mds 1394 root 318u sock 0,6 0t0 16275 can't
> identify protocol
> ceph-mds 1394 root 319u sock 0,6 0t0 16282 can't
> identify protocol
> ceph-mds 1394 root 320u sock 0,6 0t0 16284 can't
> identify protocol
> ceph-mds 1394 root 321u sock 0,6 0t0 16286 can't
> identify protocol
> ceph-mds 1394 root 322u sock 0,6 0t0 16288 can't
> identify protocol
> ceph-mds 1394 root 323u sock 0,6 0t0 16290 can't
> identify protocol
> ceph-mds 1394 root 324u sock 0,6 0t0 16297 can't
> identify protocol
> ceph-mds 1394 root 325u sock 0,6 0t0 16299 can't
> identify protocol
> ceph-mds 1394 root 326u sock 0,6 0t0 16301 can't
> identify protocol
> ceph-mds 1394 root 327u sock 0,6 0t0 16303 can't
> identify protocol
> ceph-mds 1394 root 328u sock 0,6 0t0 16305 can't
> identify protocol
> ceph-mds 1394 root 329u sock 0,6 0t0 16307 can't
> identify protocol
> ceph-mds 1394 root 330u sock 0,6 0t0 16310 can't
> identify protocol
> ceph-mds 1394 root 331u sock 0,6 0t0 16311 can't
> identify protocol
> ceph-mds 1394 root 332u sock 0,6 0t0 16313 can't
> identify protocol
> ceph-mds 1394 root 333u sock 0,6 0t0 16315 can't
> identify protocol
> ceph-mds 1394 root 334u sock 0,6 0t0 16317 can't
> identify protocol
> ceph-mds 1394 root 335u sock 0,6 0t0 18794 can't
> identify protocol
> ceph-mds 1394 root 336u sock 0,6 0t0 16346 can't
> identify protocol
> ceph-mds 1394 root 337u sock 0,6 0t0 16358 can't
> identify protocol
> ceph-mds 1394 root 338u sock 0,6 0t0 16360 can't
> identify protocol
> ceph-mds 1394 root 339u sock 0,6 0t0 16362 can't
> identify protocol
> ceph-mds 1394 root 340u sock 0,6 0t0 16370 can't
> identify protocol
> ceph-mds 1394 root 341u sock 0,6 0t0 16377 can't
> identify protocol
> ceph-mds 1394 root 342u sock 0,6 0t0 19386 can't
> identify protocol
> ceph-mds 1394 root 343u sock 0,6 0t0 19398 can't
> identify protocol
> ceph-mds 1394 root 344u sock 0,6 0t0 19435 can't
> identify protocol
> ceph-mds 1394 root 345u sock 0,6 0t0 19441 can't
> identify protocol
> ceph-mds 1394 root 346u sock 0,6 0t0 19489 can't
> identify protocol
> ceph-mds 1394 root 347u sock 0,6 0t0 19502 can't
> identify protocol
> ceph-mds 1394 root 348u sock 0,6 0t0 19519 can't
> identify protocol
> ceph-mds 1394 root 349u sock 0,6 0t0 19521 can't
> identify protocol
> ceph-mds 1394 root 350u sock 0,6 0t0 19523 can't
> identify protocol
> ceph-mds 1394 root 351u sock 0,6 0t0 19525 can't
> identify protocol
> ceph-mds 1394 root 352u sock 0,6 0t0 19527 can't
> identify protocol
> ceph-mds 1394 root 353u sock 0,6 0t0 19529 can't
> identify protocol
> ceph-mds 1394 root 354u sock 0,6 0t0 19531 can't
> identify protocol
> ceph-mds 1394 root 355u sock 0,6 0t0 19533 can't
> identify protocol
> ceph-mds 1394 root 356u sock 0,6 0t0 19535 can't
> identify protocol
> ceph-mds 1394 root 357u sock 0,6 0t0 19537 can't
> identify protocol
> ceph-mds 1394 root 358u sock 0,6 0t0 19539 can't
> identify protocol
> ceph-mds 1394 root 359u sock 0,6 0t0 19541 can't
> identify protocol
> ceph-mds 1394 root 360u sock 0,6 0t0 19543 can't
> identify protocol
> ceph-mds 1394 root 361u sock 0,6 0t0 19545 can't
> identify protocol
> ceph-mds 1394 root 362u sock 0,6 0t0 19547 can't
> identify protocol
> ceph-mds 1394 root 363u sock 0,6 0t0 19549 can't
> identify protocol
> ceph-mds 1394 root 364u sock 0,6 0t0 19551 can't
> identify protocol
> ceph-mds 1394 root 365u sock 0,6 0t0 19554 can't
> identify protocol
> ceph-mds 1394 root 366u sock 0,6 0t0 19555 can't
> identify protocol
> ceph-mds 1394 root 367u sock 0,6 0t0 19557 can't
> identify protocol
> ceph-mds 1394 root 368u sock 0,6 0t0 19559 can't
> identify protocol
> ceph-mds 1394 root 369u sock 0,6 0t0 19567 can't
> identify protocol
> ceph-mds 1394 root 370u sock 0,6 0t0 19568 can't
> identify protocol
> ceph-mds 1394 root 371u sock 0,6 0t0 19571 can't
> identify protocol
> ceph-mds 1394 root 372u sock 0,6 0t0 19572 can't
> identify protocol
> ceph-mds 1394 root 373u sock 0,6 0t0 19575 can't
> identify protocol
> ceph-mds 1394 root 374u sock 0,6 0t0 19576 can't
> identify protocol
> ceph-mds 1394 root 375u sock 0,6 0t0 19578 can't
> identify protocol
> ceph-mds 1394 root 376u sock 0,6 0t0 19580 can't
> identify protocol
> ceph-mds 1394 root 377u sock 0,6 0t0 19583 can't
> identify protocol
> ceph-mds 1394 root 378u sock 0,6 0t0 19584 can't
> identify protocol
> ceph-mds 1394 root 379u sock 0,6 0t0 19586 can't
> identify protocol
> ceph-mds 1394 root 380u sock 0,6 0t0 19588 can't
> identify protocol
> ceph-mds 1394 root 381u sock 0,6 0t0 19590 can't
> identify protocol
> ceph-mds 1394 root 382u sock 0,6 0t0 19592 can't
> identify protocol
> ceph-mds 1394 root 383u sock 0,6 0t0 19605 can't
> identify protocol
> ceph-mds 1394 root 384u sock 0,6 0t0 19607 can't
> identify protocol
> ceph-mds 1394 root 385u sock 0,6 0t0 19609 can't
> identify protocol
> ceph-mds 1394 root 386u sock 0,6 0t0 19611 can't
> identify protocol
> ceph-mds 1394 root 387u sock 0,6 0t0 19618 can't
> identify protocol
> ceph-mds 1394 root 388u sock 0,6 0t0 19620 can't
> identify protocol
> ceph-mds 1394 root 389u sock 0,6 0t0 19622 can't
> identify protocol
> ceph-mds 1394 root 390u sock 0,6 0t0 19624 can't
> identify protocol
> ceph-mds 1394 root 391u sock 0,6 0t0 19626 can't
> identify protocol
> ceph-mds 1394 root 392u sock 0,6 0t0 19628 can't
> identify protocol
> ceph-mds 1394 root 393u sock 0,6 0t0 19630 can't
> identify protocol
> ceph-mds 1394 root 394u sock 0,6 0t0 19632 can't
> identify protocol
> ceph-mds 1394 root 395u sock 0,6 0t0 19641 can't
> identify protocol
> ceph-mds 1394 root 396u sock 0,6 0t0 19642 can't
> identify protocol
> ceph-mds 1394 root 397u sock 0,6 0t0 19643 can't
> identify protocol
> ceph-mds 1394 root 398u sock 0,6 0t0 19645 can't
> identify protocol
> ceph-mds 1394 root 399u sock 0,6 0t0 19647 can't
> identify protocol
> ceph-mds 1394 root 400u sock 0,6 0t0 19649 can't
> identify protocol
> ceph-mds 1394 root 401u sock 0,6 0t0 19651 can't
> identify protocol
> ceph-mds 1394 root 402u sock 0,6 0t0 19653 can't
> identify protocol
> ceph-mds 1394 root 403u sock 0,6 0t0 19666 can't
> identify protocol
> ceph-mds 1394 root 404u sock 0,6 0t0 19667 can't
> identify protocol
> ceph-mds 1394 root 405u sock 0,6 0t0 19668 can't
> identify protocol
> ceph-mds 1394 root 406u sock 0,6 0t0 19675 can't
> identify protocol
> ceph-mds 1394 root 407u sock 0,6 0t0 19677 can't
> identify protocol
> ceph-mds 1394 root 408u sock 0,6 0t0 19679 can't
> identify protocol
> ceph-mds 1394 root 409u sock 0,6 0t0 19681 can't
> identify protocol
> ceph-mds 1394 root 410u sock 0,6 0t0 19683 can't
> identify protocol
> ceph-mds 1394 root 411u sock 0,6 0t0 19685 can't
> identify protocol
> ceph-mds 1394 root 412u sock 0,6 0t0 19687 can't
> identify protocol
> ceph-mds 1394 root 413u sock 0,6 0t0 19689 can't
> identify protocol
> ceph-mds 1394 root 414u sock 0,6 0t0 19691 can't
> identify protocol
> ceph-mds 1394 root 415u sock 0,6 0t0 19693 can't
> identify protocol
> ceph-mds 1394 root 416u sock 0,6 0t0 19695 can't
> identify protocol
> ceph-mds 1394 root 417u sock 0,6 0t0 19697 can't
> identify protocol
> ceph-mds 1394 root 418u sock 0,6 0t0 19699 can't
> identify protocol
> ceph-mds 1394 root 419u sock 0,6 0t0 19701 can't
> identify protocol
> ceph-mds 1394 root 420u sock 0,6 0t0 19704 can't
> identify protocol
> ceph-mds 1394 root 421u sock 0,6 0t0 19705 can't
> identify protocol
> ceph-mds 1394 root 422u sock 0,6 0t0 19716 can't
> identify protocol
> ceph-mds 1394 root 423u sock 0,6 0t0 19718 can't
> identify protocol
> ceph-mds 1394 root 424u sock 0,6 0t0 19707 can't
> identify protocol
> ceph-mds 1394 root 425u sock 0,6 0t0 19709 can't
> identify protocol
> ceph-mds 1394 root 426u sock 0,6 0t0 19720 can't
> identify protocol
> ceph-mds 1394 root 427u sock 0,6 0t0 19727 can't
> identify protocol
> ceph-mds 1394 root 428u sock 0,6 0t0 19735 can't
> identify protocol
> ceph-mds 1394 root 429u sock 0,6 0t0 19736 can't
> identify protocol
> ceph-mds 1394 root 430u sock 0,6 0t0 19738 can't
> identify protocol
> ceph-mds 1394 root 431u sock 0,6 0t0 19740 can't
> identify protocol
> ceph-mds 1394 root 432u sock 0,6 0t0 19742 can't
> identify protocol
> ceph-mds 1394 root 433u sock 0,6 0t0 19744 can't
> identify protocol
> ceph-mds 1394 root 434u sock 0,6 0t0 19746 can't
> identify protocol
> ceph-mds 1394 root 435u sock 0,6 0t0 19748 can't
> identify protocol
> ceph-mds 1394 root 436u sock 0,6 0t0 19750 can't
> identify protocol
> ceph-mds 1394 root 437u sock 0,6 0t0 19752 can't
> identify protocol
> ceph-mds 1394 root 438u sock 0,6 0t0 19754 can't
> identify protocol
> ceph-mds 1394 root 439u sock 0,6 0t0 19756 can't
> identify protocol
> ceph-mds 1394 root 440u sock 0,6 0t0 19765 can't
> identify protocol
> ceph-mds 1394 root 441u sock 0,6 0t0 19766 can't
> identify protocol
> ceph-mds 1394 root 442u sock 0,6 0t0 19767 can't
> identify protocol
> ceph-mds 1394 root 443u sock 0,6 0t0 19769 can't
> identify protocol
> ceph-mds 1394 root 444u sock 0,6 0t0 19771 can't
> identify protocol
> ceph-mds 1394 root 445u sock 0,6 0t0 19773 can't
> identify protocol
> ceph-mds 1394 root 446u sock 0,6 0t0 19775 can't
> identify protocol
> ceph-mds 1394 root 447u sock 0,6 0t0 19777 can't
> identify protocol
> ceph-mds 1394 root 448u sock 0,6 0t0 19784 can't
> identify protocol
> ceph-mds 1394 root 449u sock 0,6 0t0 19786 can't
> identify protocol
> ceph-mds 1394 root 450u sock 0,6 0t0 19788 can't
> identify protocol
> ceph-mds 1394 root 451u sock 0,6 0t0 19795 can't
> identify protocol
> ceph-mds 1394 root 452u sock 0,6 0t0 19797 can't
> identify protocol
> ceph-mds 1394 root 453u sock 0,6 0t0 19799 can't
> identify protocol
> ceph-mds 1394 root 454u sock 0,6 0t0 19801 can't
> identify protocol
> ceph-mds 1394 root 455u sock 0,6 0t0 19803 can't
> identify protocol
> ceph-mds 1394 root 456u sock 0,6 0t0 19811 can't
> identify protocol
> ceph-mds 1394 root 457u sock 0,6 0t0 19812 can't
> identify protocol
> ceph-mds 1394 root 458u sock 0,6 0t0 19814 can't
> identify protocol
> ceph-mds 1394 root 459u sock 0,6 0t0 19816 can't
> identify protocol
> ceph-mds 1394 root 460u sock 0,6 0t0 19819 can't
> identify protocol
> ceph-mds 1394 root 461u sock 0,6 0t0 19820 can't
> identify protocol
> ceph-mds 1394 root 462u sock 0,6 0t0 19830 can't
> identify protocol
> ceph-mds 1394 root 463u sock 0,6 0t0 19831 can't
> identify protocol
> ceph-mds 1394 root 464u sock 0,6 0t0 19832 can't
> identify protocol
> ceph-mds 1394 root 465u sock 0,6 0t0 19833 can't
> identify protocol
> ceph-mds 1394 root 466u sock 0,6 0t0 19835 can't
> identify protocol
> ceph-mds 1394 root 467u sock 0,6 0t0 19837 can't
> identify protocol
> ceph-mds 1394 root 468u sock 0,6 0t0 19848 can't
> identify protocol
> ceph-mds 1394 root 469u sock 0,6 0t0 19849 can't
> identify protocol
> ceph-mds 1394 root 470u sock 0,6 0t0 19850 can't
> identify protocol
> ceph-mds 1394 root 471u sock 0,6 0t0 19851 can't
> identify protocol
> ceph-mds 1394 root 472u sock 0,6 0t0 19852 can't
> identify protocol
> ceph-mds 1394 root 473u sock 0,6 0t0 19854 can't
> identify protocol
> ceph-mds 1394 root 474u sock 0,6 0t0 19856 can't
> identify protocol
> ceph-mds 1394 root 475u sock 0,6 0t0 19863 can't
> identify protocol
> ceph-mds 1394 root 476u sock 0,6 0t0 19865 can't
> identify protocol
> ceph-mds 1394 root 477u sock 0,6 0t0 19867 can't
> identify protocol
> ceph-mds 1394 root 478u sock 0,6 0t0 19869 can't
> identify protocol
> ceph-mds 1394 root 479u sock 0,6 0t0 19871 can't
> identify protocol
> ceph-mds 1394 root 480u sock 0,6 0t0 19878 can't
> identify protocol
> ceph-mds 1394 root 481u sock 0,6 0t0 19880 can't
> identify protocol
> ceph-mds 1394 root 482u sock 0,6 0t0 19882 can't
> identify protocol
> ceph-mds 1394 root 483u sock 0,6 0t0 19884 can't
> identify protocol
> ceph-mds 1394 root 484u sock 0,6 0t0 19886 can't
> identify protocol
> ceph-mds 1394 root 485u sock 0,6 0t0 19888 can't
> identify protocol
> ceph-mds 1394 root 486u sock 0,6 0t0 19890 can't
> identify protocol
> ceph-mds 1394 root 487u sock 0,6 0t0 19892 can't
> identify protocol
> ceph-mds 1394 root 488u sock 0,6 0t0 19894 can't
> identify protocol
> ceph-mds 1394 root 489u sock 0,6 0t0 19902 can't
> identify protocol
> ceph-mds 1394 root 490u sock 0,6 0t0 19903 can't
> identify protocol
> ceph-mds 1394 root 491u sock 0,6 0t0 19905 can't
> identify protocol
> ceph-mds 1394 root 492u sock 0,6 0t0 19907 can't
> identify protocol
> ceph-mds 1394 root 493u sock 0,6 0t0 19909 can't
> identify protocol
> ceph-mds 1394 root 494u sock 0,6 0t0 19911 can't
> identify protocol
> ceph-mds 1394 root 495u sock 0,6 0t0 19913 can't
> identify protocol
> ceph-mds 1394 root 496u sock 0,6 0t0 19915 can't
> identify protocol
> ceph-mds 1394 root 497u sock 0,6 0t0 19922 can't
> identify protocol
> ceph-mds 1394 root 498u sock 0,6 0t0 19924 can't
> identify protocol
> ceph-mds 1394 root 499u sock 0,6 0t0 19926 can't
> identify protocol
> ceph-mds 1394 root 500u sock 0,6 0t0 19934 can't
> identify protocol
> ceph-mds 1394 root 501u sock 0,6 0t0 19935 can't
> identify protocol
> ceph-mds 1394 root 502u sock 0,6 0t0 19937 can't
> identify protocol
> ceph-mds 1394 root 503u sock 0,6 0t0 19943 can't
> identify protocol
> ceph-mds 1394 root 504u sock 0,6 0t0 19951 can't
> identify protocol
> ceph-mds 1394 root 505u sock 0,6 0t0 19952 can't
> identify protocol
> ceph-mds 1394 root 506u sock 0,6 0t0 19954 can't
> identify protocol
> ceph-mds 1394 root 507u sock 0,6 0t0 19956 can't
> identify protocol
> ceph-mds 1394 root 508u sock 0,6 0t0 19958 can't
> identify protocol
> ceph-mds 1394 root 509u sock 0,6 0t0 19960 can't
> identify protocol
> ceph-mds 1394 root 510u sock 0,6 0t0 19962 can't
> identify protocol
> ceph-mds 1394 root 511u IPv4 41250 0t0 TCP
> 192.168.141.123:34406->192.168.141.129:6800 (ESTABLISHED)
> ceph-mds 1394 root 512u sock 0,6 0t0 19974 can't
> identify protocol
> ceph-mds 1394 root 513u sock 0,6 0t0 19975 can't
> identify protocol
> ceph-mds 1394 root 514u sock 0,6 0t0 19976 can't
> identify protocol
> ceph-mds 1394 root 515u sock 0,6 0t0 19977 can't
> identify protocol
> ceph-mds 1394 root 516u sock 0,6 0t0 20409 can't
> identify protocol
> ceph-mds 1394 root 517u sock 0,6 0t0 19986 can't
> identify protocol
> ceph-mds 1394 root 518u sock 0,6 0t0 19988 can't
> identify protocol
> ceph-mds 1394 root 519u sock 0,6 0t0 20446 can't
> identify protocol
> ceph-mds 1394 root 520u sock 0,6 0t0 19997 can't
> identify protocol
> ceph-mds 1394 root 521u sock 0,6 0t0 19999 can't
> identify protocol
> ceph-mds 1394 root 522u sock 0,6 0t0 20001 can't
> identify protocol
> ceph-mds 1394 root 523u sock 0,6 0t0 20003 can't
> identify protocol
> ceph-mds 1394 root 524u sock 0,6 0t0 20010 can't
> identify protocol
> ceph-mds 1394 root 525u sock 0,6 0t0 20012 can't
> identify protocol
> ceph-mds 1394 root 526u sock 0,6 0t0 20014 can't
> identify protocol
> ceph-mds 1394 root 527u sock 0,6 0t0 20022 can't
> identify protocol
> ceph-mds 1394 root 528u sock 0,6 0t0 20023 can't
> identify protocol
> ceph-mds 1394 root 529u sock 0,6 0t0 20025 can't
> identify protocol
> ceph-mds 1394 root 530u sock 0,6 0t0 20032 can't
> identify protocol
> ceph-mds 1394 root 531u sock 0,6 0t0 20039 can't
> identify protocol
> ceph-mds 1394 root 532u sock 0,6 0t0 20046 can't
> identify protocol
> ceph-mds 1394 root 533u sock 0,6 0t0 20110 can't
> identify protocol
> ceph-mds 1394 root 534u sock 0,6 0t0 20134 can't
> identify protocol
> ceph-mds 1394 root 535u sock 0,6 0t0 20146 can't
> identify protocol
> ceph-mds 1394 root 536u sock 0,6 0t0 20158 can't
> identify protocol
> ceph-mds 1394 root 537u sock 0,6 0t0 20170 can't
> identify protocol
> ceph-mds 1394 root 538u sock 0,6 0t0 20192 can't
> identify protocol
> ceph-mds 1394 root 539u sock 0,6 0t0 40930 can't
> identify protocol
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 19:05 ` Noah Watkins
2011-12-05 19:11 ` Sage Weil
@ 2011-12-05 19:36 ` Tommi Virtanen
2011-12-07 5:49 ` Gregory Farnum
1 sibling, 1 reply; 12+ messages in thread
From: Tommi Virtanen @ 2011-12-05 19:36 UTC (permalink / raw)
To: Noah Watkins; +Cc: Noah Watkins, Sage Weil, ceph-devel
On Mon, Dec 5, 2011 at 11:05, Noah Watkins <jayhawk@cs.ucsc.edu> wrote:
> root@issdm-23:~# lsof -n -p 1394
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> ceph-mds 1394 root 12u sock 0,6 0t0 11560 can't
> identify protocol
There's 506 of these guys in the listing. I would say there's a leak
of sockets that get created with socket(2), but don't get bind(2) or
connect(2) called on them.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-05 19:36 ` Tommi Virtanen
@ 2011-12-07 5:49 ` Gregory Farnum
2011-12-07 22:33 ` Gregory Farnum
0 siblings, 1 reply; 12+ messages in thread
From: Gregory Farnum @ 2011-12-07 5:49 UTC (permalink / raw)
To: Tommi Virtanen; +Cc: Noah Watkins, Sage Weil, ceph-devel
Looks like these are all sockets that don't get closed because we send
the associated Pipes into reconnect, but they never actually do so
(and never move on out of it into shutting down). We made some changes
to how reconnect and cleanup are done that probably weren't
complete/missed a case. (Maybe only clients actually try to reconnect,
but everybody goes into that state?)
For tomorrow!
On Mon, Dec 5, 2011 at 11:36 AM, Tommi Virtanen
<tommi.virtanen@dreamhost.com> wrote:
> On Mon, Dec 5, 2011 at 11:05, Noah Watkins <jayhawk@cs.ucsc.edu> wrote:
>> root@issdm-23:~# lsof -n -p 1394
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> ceph-mds 1394 root 12u sock 0,6 0t0 11560 can't
>> identify protocol
>
> There's 506 of these guys in the listing. I would say there's a leak
> of sockets that get created with socket(2), but don't get bind(2) or
> connect(2) called on them.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-07 5:49 ` Gregory Farnum
@ 2011-12-07 22:33 ` Gregory Farnum
2011-12-09 0:40 ` Noah Watkins
0 siblings, 1 reply; 12+ messages in thread
From: Gregory Farnum @ 2011-12-07 22:33 UTC (permalink / raw)
To: Noah Watkins; +Cc: ceph-devel
Noah:
Branch wip-messenger contains the patch you want (plus one or two
cleanups), or apply the below. Should handle it, although I'd like to
do a more thorough cleanup of this problem at a later time.
-Greg
From 8c4f4748e8b683f5b4ea939295793421c0ab7b61 Mon Sep 17 00:00:00 2001
From: Greg Farnum <gregory.farnum@dreamhost.com>
Date: Wed, 7 Dec 2011 14:19:23 -0800
Subject: [PATCH] mds: mark_disposable when closing a Client connection.
This is causing issues since the Client's ack of the MClientSession
is somehow not getting back to the MDS. We should make the
Client/SimpleMessenger be better-behaved about shutdown, but for
now this solution matches how we're working elsewhere in our codebase.
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
---
src/mds/Server.cc | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/src/mds/Server.cc b/src/mds/Server.cc
index a9ef64f..d44bd7e 100644
--- a/src/mds/Server.cc
+++ b/src/mds/Server.cc
@@ -289,6 +289,7 @@ void Server::_session_logged(Session *session,
uint64_t state_seq, bool open, ve
// reset session
mds->send_message_client(new
MClientSession(CEPH_SESSION_CLOSE), session);
mds->sessionmap.set_state(session, Session::STATE_CLOSED);
+ mds->messenger->mark_disposable(session->connection);
session->clear();
} else if (session->is_killing()) {
// destroy session, close connection
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: Client receives 'connection refused' only after heavy use
2011-12-07 22:33 ` Gregory Farnum
@ 2011-12-09 0:40 ` Noah Watkins
0 siblings, 0 replies; 12+ messages in thread
From: Noah Watkins @ 2011-12-09 0:40 UTC (permalink / raw)
To: Gregory Farnum; +Cc: Noah Watkins, ceph-devel
This patch appears to be working well!
Thanks,
Noah
On 12/07/2011 02:33 PM, Gregory Farnum wrote:
> Noah:
> Branch wip-messenger contains the patch you want (plus one or two
> cleanups), or apply the below. Should handle it, although I'd like to
> do a more thorough cleanup of this problem at a later time.
> -Greg
>
> From 8c4f4748e8b683f5b4ea939295793421c0ab7b61 Mon Sep 17 00:00:00 2001
> From: Greg Farnum<gregory.farnum@dreamhost.com>
> Date: Wed, 7 Dec 2011 14:19:23 -0800
> Subject: [PATCH] mds: mark_disposable when closing a Client connection.
>
> This is causing issues since the Client's ack of the MClientSession
> is somehow not getting back to the MDS. We should make the
> Client/SimpleMessenger be better-behaved about shutdown, but for
> now this solution matches how we're working elsewhere in our codebase.
>
> Signed-off-by: Greg Farnum<gregory.farnum@dreamhost.com>
> ---
> src/mds/Server.cc | 1 +
> 1 files changed, 1 insertions(+), 0 deletions(-)
>
> diff --git a/src/mds/Server.cc b/src/mds/Server.cc
> index a9ef64f..d44bd7e 100644
> --- a/src/mds/Server.cc
> +++ b/src/mds/Server.cc
> @@ -289,6 +289,7 @@ void Server::_session_logged(Session *session,
> uint64_t state_seq, bool open, ve
> // reset session
> mds->send_message_client(new
> MClientSession(CEPH_SESSION_CLOSE), session);
> mds->sessionmap.set_state(session, Session::STATE_CLOSED);
> + mds->messenger->mark_disposable(session->connection);
> session->clear();
> } else if (session->is_killing()) {
> // destroy session, close connection
>
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2011-12-09 0:40 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-05 0:22 Client receives 'connection refused' only after heavy use Noah Watkins
2011-12-05 1:03 ` Noah Watkins
2011-12-05 4:47 ` Sage Weil
2011-12-05 6:22 ` Noah Watkins
2011-12-05 18:57 ` Noah Watkins
2011-12-05 19:03 ` Tommi Virtanen
2011-12-05 19:05 ` Noah Watkins
2011-12-05 19:11 ` Sage Weil
2011-12-05 19:36 ` Tommi Virtanen
2011-12-07 5:49 ` Gregory Farnum
2011-12-07 22:33 ` Gregory Farnum
2011-12-09 0:40 ` Noah Watkins
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.