From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 362E854652 for ; Tue, 9 Jun 2026 09:59:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780999170; cv=none; b=c8Idh8xqrwl7RL1dblZn3siR1bO23Y8QUX+qc4cSS3CLESkxBvFWA3cR2zalctixnMD2d2VlBgmEewihueFaI8Y0lgVIsu0YvkQrM0zR/LPPd2+ol+YHDciRePjliqzJqfvtWz9NViOv9htLDa+2jcwe0RL3wAvoWyRPSMRedrc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780999170; c=relaxed/simple; bh=hMMLeqXr1Gd02Wmmux5nk8+zyoHUnGPcletJMA3nMK8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gGgGR91Gj+3yPPJmeFAgycNX6og3x0s5dTwSug9RvU7cWVJX/K2u3iQxQckPCeqS97ZkAX9JTcTpPJo67q5q2YQEYtqIAjpht4tJQvDgEShY/EtopvP15QGl2I0RN8lmx0LZfaZ+9FYrVnIFWG0iiLu0j4bE11D+NR2rqtoFnbU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=R/7iFyku; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=WA1MKHtY; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=R/7iFyku; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=WA1MKHtY; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="R/7iFyku"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="WA1MKHtY"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="R/7iFyku"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="WA1MKHtY" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 50500758F4; Tue, 9 Jun 2026 09:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1780999167; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=s09ut5zCsbLHcdVBJEOkklCvvp/9ydqG9Pd7oNFwddE=; b=R/7iFykuUabeginOMThm7KIuj7ub8JvoZz9u/J0hsntatVbhGd2+guPoVj/DsZsZ/WTm40 4DnFwwxjX8pqlsh6vKUatI1WyjgBpDwfaJFVZMwnYgGc9ox4sOP+iUZ7xSqnBn/YIeM++0 GUDKxBFxgXiALkQUDsHRQjMg559gDjQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1780999167; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=s09ut5zCsbLHcdVBJEOkklCvvp/9ydqG9Pd7oNFwddE=; b=WA1MKHtYolUELcET7nco0OnQUbekh2TxWUWdJc+MHdIQpFEMWJu/VHMYZZVQLfqdAfJDez 2Yj/TlAFHjvNmbBg== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b="R/7iFyku"; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=WA1MKHtY DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1780999167; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=s09ut5zCsbLHcdVBJEOkklCvvp/9ydqG9Pd7oNFwddE=; b=R/7iFykuUabeginOMThm7KIuj7ub8JvoZz9u/J0hsntatVbhGd2+guPoVj/DsZsZ/WTm40 4DnFwwxjX8pqlsh6vKUatI1WyjgBpDwfaJFVZMwnYgGc9ox4sOP+iUZ7xSqnBn/YIeM++0 GUDKxBFxgXiALkQUDsHRQjMg559gDjQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1780999167; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=s09ut5zCsbLHcdVBJEOkklCvvp/9ydqG9Pd7oNFwddE=; b=WA1MKHtYolUELcET7nco0OnQUbekh2TxWUWdJc+MHdIQpFEMWJu/VHMYZZVQLfqdAfJDez 2Yj/TlAFHjvNmbBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 360A7779A7; Tue, 9 Jun 2026 09:59:27 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id WyYnDf/jJ2r7PwAAD6G6ig (envelope-from ); Tue, 09 Jun 2026 09:59:27 +0000 Received: by incl.suse.cz (Postfix, from userid 1000) id D81C310A50D; Tue, 9 Jun 2026 11:59:26 +0200 (CEST) Date: Tue, 9 Jun 2026 11:59:26 +0200 From: Jiri Wiesner To: Dimitri Sivanich Cc: Thomas Gleixner , Linux Kernel Mailing List , Steve Wahl , Justin Ernst , Kyle Meyer , Russ Anderson , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , "Peter Zijlstra (Intel)" , Ilpo =?iso-8859-1?Q?J=E4rvinen?= , Marco Elver , "Guilherme G. Piccoli" , Nikunj A Dadhania , "Xin Li (Intel)" , Dimitri Sivanich Subject: Re: [PATCH v4 0/2] x86/tsc: Exempt recent UV systems from clocksource watchdog checks to avoid false positives. Message-ID: References: <87lddcv9vd.ffs@tglx> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Action: no action X-Rspamd-Queue-Id: 50500758F4 X-Spam-Flag: NO X-Spam-Score: -4.01 X-Spam-Level: X-Spamd-Result: default: False [-4.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo,suse.de:dkim]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[19]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_THREE(0.00)[3]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[suse.de:dkim,imap1.dmz-prg2.suse.org:rdns,imap1.dmz-prg2.suse.org:helo]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCVD_TLS_LAST(0.00)[]; FUZZY_RATELIMITED(0.00)[rspamd.com]; DKIM_TRACE(0.00)[suse.de:+] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org On Thu, May 21, 2026 at 09:08:34PM -0500, Dimitri Sivanich wrote: > On Thu, May 21, 2026 at 09:30:14PM +0200, Thomas Gleixner wrote: > > On Thu, May 21 2026 at 08:17, Dimitri Sivanich wrote: > > > HPE UV hardware and firmware is designed to ensure a reliable and > > > synchronized TSC mechanism. Comparing the TSC against secondary > > > clocksources can result in false positives due to variable access > > > latency caused by system traffic. I do not think that the access latency of the reference clocksource, sgi_rtc in this case, is the cause of the false positives. I think sgi_rtc really experiences time skew. More details below. > The best course of action against > > > these false positives has been found to simply disable watchdog > > > checking of the TSC. > > > > > > Commits [1] and [2] were introduced to avoid an issue where the TSC > > > is falsely declared unstable by exempting qualified platforms of up > > > to 4-sockets from TSC clocksource watchdog checking. Extend that > > > exemption to include recent and future UV platforms. > > > > Jiri asked you in the V3 submission: > > > > "A new implementation of the clocksource watchdog has been merged into > > the upstream kernel. One of the changes made by the new clocksource > > watchdog implementation is that reference clocksource reads are made > > on the boot CPU only. Perhaps, the sgi_rtc clocksource would work well > > with this implementation. So, testing is needed in order to find out > > if this patch are any future in the upstream Linux. Dimitri, would you > > be able to run tests on UV systems to check if the new clocksource > > watchdog implementation works and the hardware limitations of sgi_rtc > > do not get in the way?" > > > > This question is still not answered by you and it has been confirmed > > that the new watchdog works flawlessly on a 1920 threads 16 socket > > system under massive load and system traffic. > > I tested a 7.1-rc4 kernel on a 2048 thread 16 socket system and, while > under test, the TSC did get marked as unstable after a series of "sgi_rtc > read timed out" warnings. The new clocksource watchdog implementation makes sure to act on time skew only if the time between two reference clocksource readouts does not exceed 50 us. The threshold for evaluating time skew (based on SHIFT_500PPM) is 244 us for a 500 ms interval plus the measured reference clocksource readout latency. If the comparison to the reference clocksource fails on CPU 0 the time skew between the clocksource being checked and the reference clocksource must be at least 244 us. The clocksource watchdog cannot distiguish which of the clocksources is skewed, and it must make the assumption that the clocksource being checked is skewed. In the past, I worked on a bug where a customer with an HPE UV machine reported degraded performance and switches to the HPET. This kernel had the old clocksource watchdog implementation. I created a debugging kernel with the HPET as a second watchdog (not affecting the decisions by the watchdog) and got this result: > clocksource: timekeeping watchdog on CPU118: Marking clocksource 'tsc' as unstable because the skew is too large: > clocksource: 'sgi_rtc' wd_nsec: 511302794 wd_now: 1cb50e4c4b wd_last: 1ca7097111 mask: ffffffffffffff > clocksource: 'hpet' wd2_nsec: 512005960 wd2_now: 65892719 wd2_last: 64c5d684 mask: ffffffff > clocksource: 'tsc' cs_nsec: 512006458 cs_now: 86b5982cb1 cs_last: 867581bbab mask: ffffffffffffffff > clocksource: 'tsc' skewed 703664 ns (0 ms) over watchdog 'sgi_rtc' interval of 511302794 ns (511 ms) > clocksource: 'tsc' is current clocksource. > tsc: Marking TSC unstable due to clocksource watchdog > clocksource: Checking clocksource tsc synchronization from CPU 610 to CPUs 0-609,611-767. > clocksource: Switched to clocksource sgi_rtc The intervals measured by the TSC and the HPET match very well; the sgi_rtc is off. I find it hard to believe that both the TSC and the HPET would be skewed - both reporting a longer interval - while sgi_rtc was correct. I think sgi_rtc was skewed. There are several solution to work around the hardware limitation of sgi_rtc: 1. Disable the clocksource watchdog 2. Decrease the rating of sgi_rtc 3. Disable sgi_rtc Solution 3 in the form of a nouvrtc parameter was previously rejected on this mailing list. The disadvantage of the solution is that each customer would have to pass the nouvrtc parameter to the kernel to avoid false positives by the clocksource watchdog, which makes no sense from the POV of OS support (as done by e.g. SUSE). -- Jiri Wiesner SUSE Labs