From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A9B12135C0 for ; Fri, 10 Jan 2025 17:10:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736529060; cv=none; b=qHhG9QG6dGQxCct9wyszddfaxObCluz9i202jK6+Lv1feArI3WXK7weK2yGkQ66gby+5I6yKXs/8kuzKr/t1K7nFBkL3gX+SUzVdBX1pGpu3w6pwl+o8WIC+c5yRu5xI2/UUd29SKs0UuMUlvtSKW4jAiV67BED71w2YdVxgwY4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736529060; c=relaxed/simple; bh=kVfLymHxVpByejOyXDJjG4VlKcBwlrYK8ybhTYIFYgc=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=UczhFid0ksNfMzJabJhglCg1t/kqebgX5Quiu42nlaxa8G+CumiYEAeQGuAxSm9CxddbVG5LOa5K2Cmn9pYKB31i7zN2b003DMFLtcdI/NIBAvaUHyagwSVy2hrxsSBXmxbg4Ra2G92RLp8QVBxjI4XMVrBYBEiYJFr0NTm0XCo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=O/aAFOIa; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="O/aAFOIa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736529057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=RIXoBUDd7J/x0EK3BcpH/gtTFxas7hY3KyqlHXWzMDQ=; b=O/aAFOIa42BCsj7d6apjmzDrNGeq2KEMgAmr0WazZHQCrYH2LwV4FNzVicxng46Ts3Mt42 6kmZXP8+qRPEkRL6f6hHQEgwHTKo3xUeACRz6/5yJGJXoC9NDlm6wpJCJU/W6ISsEAvVEB Q0hahcBV+xyE7wiEcRfCKAjUXds8UxI= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-552-PS-I0oEcOBS-q-aA1A8WAg-1; Fri, 10 Jan 2025 12:10:51 -0500 X-MC-Unique: PS-I0oEcOBS-q-aA1A8WAg-1 X-Mimecast-MFC-AGG-ID: PS-I0oEcOBS-q-aA1A8WAg Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 299B21953944; Fri, 10 Jan 2025 17:10:49 +0000 (UTC) Received: from oldenburg.str.redhat.com (unknown [10.2.16.2]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5179530001BE; Fri, 10 Jan 2025 17:10:44 +0000 (UTC) From: Florian Weimer To: Mathieu Desnoyers Cc: Peter Zijlstra , libc-alpha@sourceware.org , "carlos@redhat.com" , Mark Rutland , linux-kernel , x86@kernel.org, paulmck , Michael Jeanson Subject: Re: Prevent inconsistent CPU state after sequence of dlclose/dlopen In-Reply-To: <8c1ad304-61bb-4bdf-aa75-8633f3d0196c@efficios.com> (Mathieu Desnoyers's message of "Fri, 10 Jan 2025 12:02:27 -0500") References: <20250110165412.GC4213@noisy.programming.kicks-ass.net> <8c1ad304-61bb-4bdf-aa75-8633f3d0196c@efficios.com> Date: Fri, 10 Jan 2025 18:10:42 +0100 Message-ID: <87ldvitx0t.fsf@oldenburg.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 * Mathieu Desnoyers: > On 2025-01-10 11:54, Peter Zijlstra wrote: >> On Fri, Jan 10, 2025 at 10:55:36AM -0500, Mathieu Desnoyers wrote: >>> Hi, >>> >>> I was discussing with Mark Rutland recently, and he pointed out that a >>> sequence of dlclose/dlopen mapping new code at the same addresses in >>> multithreaded environments is an issue on ARM, and possibly on Intel/AMD >>> with the newer TLB broadcast maintenance. >> What is the exact race? Should not munmap() invalidate the TLBs >> before >> it allows overlapping mmap() to complete? > > The race Mark mentioned (on ARM) is AFAIU the following scenario: > > CPU 0 CPU 1 > > - dlopen() > - mmap PROT_EXEC @addr > - fetch insn @addr, CPU state expects unchanged insn. > - execute unrelated code > - dlclose(addr) > - munmap @addr > - dlopen() > - mmap PROT_EXEC @addr > - fetch new insn @addr. Incoherent CPU state. Unmapping an object while code is executing in it is undefined. We have a problem with things like pthread_atfork handlers. We can't use locking there because fork handlers are expected to perform ample locking themselves, and an extra lock around them would run into lock ordering issues. (We tried for unrelated reasons and saw deadlocks in applications.) What we can do is bump a reference counter while we run a pthread_atfork callback (we already associate them with DSOs) and skip the munmap part in dlclose if the counter is not zero. We can complete the unmapping after the fork handler returns (maybe in the parent only). There might be other callbacks besides fork handlers that have this problem. A similar treatment is possible for some of them, hopefully all of them in glibc. We cannot cover things like std::shared_ptr destructor calls, though. But adding more barriers won't fix those, either. Thanks, Florian