From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE9B8E7E63F for ; Tue, 26 Sep 2023 14:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233558AbjIZOUt (ORCPT ); Tue, 26 Sep 2023 10:20:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233033AbjIZOUs (ORCPT ); Tue, 26 Sep 2023 10:20:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77231EB for ; Tue, 26 Sep 2023 07:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695737993; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=CLfoTuiOlkq71LtQIM368ZbzJBNnHFUnsO5iHo6yhEM=; b=WRaP8R3LbN/diCUfhdY0tbAHpbiBdmOUQwLrBq28Ysi2djVVE+zN5z80+E+dK4hmGyD3PF 1zn/TTSvywXsbBW1xjfrP7di8nJmfb/lZQ6Gjb4Dp/OAGe2uqv2AAP6d+HGNp5aLgtj5Fy yo9JmREqrgm0zwGwRZBGHIFsvVeHUHw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-96-Za_WVKlVMr-fkse-WyoN-g-1; Tue, 26 Sep 2023 10:19:50 -0400 X-MC-Unique: Za_WVKlVMr-fkse-WyoN-g-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 906C03C1ACE4; Tue, 26 Sep 2023 14:19:49 +0000 (UTC) Received: from oldenburg.str.redhat.com (unknown [10.2.16.53]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A1F78492C37; Tue, 26 Sep 2023 14:19:47 +0000 (UTC) From: Florian Weimer To: Miklos Szeredi Cc: Miklos Szeredi , Christian Brauner , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, linux-man@vger.kernel.org, linux-security-module@vger.kernel.org, Karel Zak , Ian Kent , David Howells , Al Viro , Christian Brauner , Amir Goldstein Subject: Re: [RFC PATCH 2/3] add statmnt(2) syscall References: <20230913152238.905247-1-mszeredi@redhat.com> <20230913152238.905247-3-mszeredi@redhat.com> <20230914-salzig-manifest-f6c3adb1b7b4@brauner> <20230914-lockmittel-verknallen-d1a18d76ba44@brauner> <20230918-grafik-zutreffen-995b321017ae@brauner> <871qeloxj0.fsf@oldenburg.str.redhat.com> Date: Tue, 26 Sep 2023 16:19:46 +0200 In-Reply-To: (Miklos Szeredi's message of "Tue, 26 Sep 2023 16:06:05 +0200") Message-ID: <87wmwdnhj1.fsf@oldenburg.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: linux-api@vger.kernel.org * Miklos Szeredi: >> Try-and-resize interfaces can be quite bad for data obtained from the >> network. > > In this particular case it's all local information. That's good. >> If the first call provides the minimum buffer size (like >> getgroups, but unlike readlink or the glibc *_r interfaces for NSS), >> this could at least allow us to avoid allocating too much. In >> userspace, we cannot reduce the size of the heap allocation without >> knowing where the pointers are and what they mean. > > Does it matter if the heap allocation is say 32k instead of 589bytes? > The returned strings are not limited in size, but are quite unlikely > to be over PATH_MAX. It matters if the application needs to keep a copy. > E.g. getdents apparently uses 32k buffers, which is really a tiny > amount of heap these days, but more than enough for the purpose. Not > sure if this is hard coded into libc or if it's the result of some > heuristic based on available memory, but I don't see why similar > treatment couldn't be applied to the statmount(2) syscall. getdents gets away with this buffer size because applications can copy out all the data from struct dirent if they need long-term storage. They have to do that because the usual readdir interface overwrites the buffer, potentially at the next readdir call. This means the buffer size does not introduce an amount of memory fragmention that is dependent on the directory size. With an opaque, pointer-carrying struct, copying out the data is not possible in a generic fashion. Only the parts that the application knows about can be copied out. So I think it's desirable to have a fairly exact allocation. >> I also don't quite understand the dislike of variable-sized records. >> Don't getdents, inotify, Netlink all use them? And I think at least for >> Netlink, more stuff is added all the time? > > What do you mean by variable sized records? Iterating through d_reclen-sized subojects (for getdents). Thanks, Florian