"clocks not synchronized" warning message

"clocks not synchronized" warning message

[Thomas Mueller]

hi

running ceph -w I see many of those warnings flying by:

10.07.08_14:36:49.897108   log 10.07.08_14:36:48.530057 mon1 192.168.1.52:6789/0 340 : [WRN] lease_expire from mon0 was sent from future time 10.07.08_14:36:48.886876, clocks not synchronized
10.07.08_14:36:49.897108   log 10.07.08_14:36:48.530076 mon1 192.168.1.52:6789/0 341 : [WRN] 

i've stopped ceph, ran ntpdate on every node, started ceph 
- the message is still there. 

date from the nodes:
# for i in node001 node002 node003 node004; do ssh root@$i date; done
Thu Jul  8 14:52:20 CEST 2010
Thu Jul  8 14:52:20 CEST 2010
Thu Jul  8 14:52:20 CEST 2010
Thu Jul  8 14:52:21 CEST 2010


this is on ceph.git/unstable from today.

- Thomas



ceph.conf:


[global]
	; enable secure authentication
	; auth supported = cephx

[mon]
	mon data = /ceph/mon$id

	; some minimal logging (just message traffic) to aid debugging
	;debug ms = 1

[mon0]
	host = node001
	mon addr = 192.168.1.51:6789

[mon1]
        host = node002
        mon addr = 192.168.1.52:6789

[mon2]
        host = node003
        mon addr = 192.168.1.53:6789

[mds]
	; where the mds keeps it's secret encryption keys
	keyring = /ceph/keyring.$name

[mds.node001]
	host = node001

[mds.node002]
	host = node002

[mds.node003]
        host = node003

; osd
;  You need at least one.  Two if you want data to be replicated.
;  Define as many as you like.
[osd]
	; This is where the btrfs volume will be mounted.
	osd data = /ceph/osd$id

	; Ideally, make this a separate disk or partition.  A few GB
 	; is usually enough; more if you have fast disks.  You can use
 	; a file under the osd data dir if need be
 	; (e.g. /ceph/osd$id/journal), but it will be slower than a
 	; separate disk or partition.
	osd journal = /ceph/osd$id/journal
	osd journal size = 100

[osd0]
	host = node001

	; if 'btrfs devs' is not specified, you're responsible for
	; setting up the 'osd data' dir.  if it is not btrfs, things
	; will behave up until you try to recover from a crash (which
	; usually fine for basic testing).
	;btrfs devs = /dev/sdx

[osd1]
	host = node002
	;btrfs devs = /dev/sdy

[osd2]
	host = node003
	;btrfs devs = /dev/sdx

[osd3]
	host = node004
	;btrfs devs = /dev/sdy

Re: "clocks not synchronized" warning message

[Wido den Hollander]

Hi,

I'm experiencing the same. The default "mon lease wiggle room" is set
t .001, which gives the messages in my setup.

Right now i've set the room to 0.5, which gets rid of the messages, but
obviously this is imho not the way to go.

My nodes are running openntpd and a cron which runs ntpdate every hour.

-- 
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Thu, 2010-07-08 at 12:54 +0000, Thomas Mueller wrote:
> hi
> 
> running ceph -w I see many of those warnings flying by:
> 
> 10.07.08_14:36:49.897108   log 10.07.08_14:36:48.530057 mon1 192.168.1.52:6789/0 340 : [WRN] lease_expire from mon0 was sent from future time 10.07.08_14:36:48.886876, clocks not synchronized
> 10.07.08_14:36:49.897108   log 10.07.08_14:36:48.530076 mon1 192.168.1.52:6789/0 341 : [WRN] 
> 
> i've stopped ceph, ran ntpdate on every node, started ceph 
> - the message is still there. 
> 
> date from the nodes:
> # for i in node001 node002 node003 node004; do ssh root@$i date; done
> Thu Jul  8 14:52:20 CEST 2010
> Thu Jul  8 14:52:20 CEST 2010
> Thu Jul  8 14:52:20 CEST 2010
> Thu Jul  8 14:52:21 CEST 2010
> 
> 
> this is on ceph.git/unstable from today.
> 
> - Thomas
> 
> 
> 
> ceph.conf:
> 
> 
> [global]
> 	; enable secure authentication
> 	; auth supported = cephx
> 
> [mon]
> 	mon data = /ceph/mon$id
> 
> 	; some minimal logging (just message traffic) to aid debugging
> 	;debug ms = 1
> 
> [mon0]
> 	host = node001
> 	mon addr = 192.168.1.51:6789
> 
> [mon1]
>         host = node002
>         mon addr = 192.168.1.52:6789
> 
> [mon2]
>         host = node003
>         mon addr = 192.168.1.53:6789
> 
> [mds]
> 	; where the mds keeps it's secret encryption keys
> 	keyring = /ceph/keyring.$name
> 
> [mds.node001]
> 	host = node001
> 
> [mds.node002]
> 	host = node002
> 
> [mds.node003]
>         host = node003
> 
> ; osd
> ;  You need at least one.  Two if you want data to be replicated.
> ;  Define as many as you like.
> [osd]
> 	; This is where the btrfs volume will be mounted.
> 	osd data = /ceph/osd$id
> 
> 	; Ideally, make this a separate disk or partition.  A few GB
>  	; is usually enough; more if you have fast disks.  You can use
>  	; a file under the osd data dir if need be
>  	; (e.g. /ceph/osd$id/journal), but it will be slower than a
>  	; separate disk or partition.
> 	osd journal = /ceph/osd$id/journal
> 	osd journal size = 100
> 
> [osd0]
> 	host = node001
> 
> 	; if 'btrfs devs' is not specified, you're responsible for
> 	; setting up the 'osd data' dir.  if it is not btrfs, things
> 	; will behave up until you try to recover from a crash (which
> 	; usually fine for basic testing).
> 	;btrfs devs = /dev/sdx
> 
> [osd1]
> 	host = node002
> 	;btrfs devs = /dev/sdy
> 
> [osd2]
> 	host = node003
> 	;btrfs devs = /dev/sdx
> 
> [osd3]
> 	host = node004
> 	;btrfs devs = /dev/sdy
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "clocks not synchronized" warning message

[Thomas Mueller]

Am Thu, 08 Jul 2010 16:19:28 +0200 schrieb Wido den Hollander:

> Hi,
> 
> I'm experiencing the same. The default "mon lease wiggle room" is set t
> .001, which gives the messages in my setup.

this is what I just found too - in unstable the default is now 0.010.  
with 0.5 the message was still there. with 1.0 it was away.

I think 0.010 is still a bit small for a system not connected to a atomic 
clock? :)



> 
> Right now i've set the room to 0.5, which gets rid of the messages, but
> obviously this is imho not the way to go.
> 
> My nodes are running openntpd and a cron which runs ntpdate every hour.

openntpd and ntpdate? IMHO if you are running the daemon ntpdate can't run 
- at least with the debian ntp package.


- Thomas

Re: "clocks not synchronized" warning message

[Sage Weil]

On Thu, 8 Jul 2010, Thomas Mueller wrote:

> Am Thu, 08 Jul 2010 16:19:28 +0200 schrieb Wido den Hollander:
> 
> > Hi,
> > 
> > I'm experiencing the same. The default "mon lease wiggle room" is set t
> > .001, which gives the messages in my setup.
> 
> this is what I just found too - in unstable the default is now 0.010.  
> with 0.5 the message was still there. with 1.0 it was away.
> 
> I think 0.010 is still a bit small for a system not connected to a atomic 
> clock? :)

How about .5 seconds?  We sync our machines to a local ntp server, so the 
clocks get really close (sub-ms).  I forget that NTP isn't so accurate 
over a WAN.

The issue is that the monitor leases and timeouts are something like 2 
seconds (so that failures/unavailability) are detected quickly, so the 
tolerable clock drift can't get too close to that.

sage

Re: "clocks not synchronized" warning message

[Thomas Mueller]

Am Thu, 08 Jul 2010 07:53:00 -0700 schrieb Sage Weil:

> On Thu, 8 Jul 2010, Thomas Mueller wrote:
> 
>> Am Thu, 08 Jul 2010 16:19:28 +0200 schrieb Wido den Hollander:
>> 
>> > Hi,
>> > 
>> > I'm experiencing the same. The default "mon lease wiggle room" is set
>> > t .001, which gives the messages in my setup.
>> 
>> this is what I just found too - in unstable the default is now 0.010.
>> with 0.5 the message was still there. with 1.0 it was away.
>> 
>> I think 0.010 is still a bit small for a system not connected to a
>> atomic clock? :)
> 
> How about .5 seconds?  We sync our machines to a local ntp server, so
> the clocks get really close (sub-ms).  I forget that NTP isn't so
> accurate over a WAN.
> 
> The issue is that the monitor leases and timeouts are something like 2
> seconds (so that failures/unavailability) are detected quickly, so the
> tolerable clock drift can't get too close to that.
> 
> sage

ok, if this is the problem, then I would also recommend running an ntp 
daemon (not ntpdate) on every node. A running ntpd will manage the 
timesource to be in sync. 

If anyone likes to run an mds/mon on a VM I expect problems - virtualized 
clocksources are far less stable (my experience).

- Thomas 

Re: "clocks not synchronized" warning message

[Thomas Mueller]

>> 
>> How about .5 seconds?  We sync our machines to a local ntp server, so
>> the clocks get really close (sub-ms).  I forget that NTP isn't so
>> accurate over a WAN.
>> 
>> The issue is that the monitor leases and timeouts are something like 2
>> seconds (so that failures/unavailability) are detected quickly, so the
>> tolerable clock drift can't get too close to that.
>> 
>> sage
> 
> ok, if this is the problem, then I would also recommend running an ntp
> daemon (not ntpdate) on every node. A running ntpd will manage the
> timesource to be in sync.
> 
> If anyone likes to run an mds/mon on a VM I expect problems -
> virtualized clocksources are far less stable (my experience).
> 

started the ntpd on the nodes. now i'm able to run it with "mon lease 
wiggle room = 0.1".

as Wido is using ntpdate and he's successfull with 0.5 I would say 0.5 is 
possible a good default.  

- Thomas