From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D91BFC433FE for ; Thu, 20 Oct 2022 11:55:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5DDA141BDC; Thu, 20 Oct 2022 11:55:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 5DDA141BDC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id in0oeFjt9FP7; Thu, 20 Oct 2022 11:55:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 1A51F41D8E; Thu, 20 Oct 2022 11:55:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 1A51F41D8E Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id B65B61BF287 for ; Thu, 20 Oct 2022 11:55:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 920756FA08 for ; Thu, 20 Oct 2022 11:55:17 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 920756FA08 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ynsaci2ACr8Z for ; Thu, 20 Oct 2022 11:55:16 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B6F6A60AB7 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by smtp3.osuosl.org (Postfix) with ESMTPS id B6F6A60AB7 for ; Thu, 20 Oct 2022 11:55:16 +0000 (UTC) Received: by mail-wm1-x332.google.com with SMTP id v11so4738855wmd.1 for ; Thu, 20 Oct 2022 04:55:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nl6gBuDkdBBhlJUsdoEyoRtZB7CmnWhJWX2Ky56AqZ4=; b=2cfUNc1K25Ve6DEL2aRp9tomc4gb1MPf8H2/8yRBr/hAnvj8pHLQfYJ+xjU21XMkod Ednqmgx4Gk8t3/IznX/Z41ufaRwEAODRt7Z99nQdknncFwAYI1Ig4dt67kwEx9dSnb2Q 5ffOEY46D9bSX/ZRxmCNa9Lm+vm2VrJybxcDDfTe04OTSMpN7FoR8EVW1PZp/iy5lxwQ 50LQ9yeevUaPXZ3puS0fxuEXwPtgeVjBh56laVnl5Z7EvaU5llP79zPNKP8zmiipus8m xTsujyyATzb+T+kwzKpXmbp6MzvbvtfPTKDxC3SD86D0fUE/F3xXpoP1Z7zLYqRH7B3S 5mUg== X-Gm-Message-State: ACrzQf10XQvaNCt7Xpp2cTTgk7Ytp5qHtsbt1p+27H9tyj0QtQW6ew6J 3LEucH3qDmRb7TF2gXEZORL3d53cYQs= X-Google-Smtp-Source: AMsMyM4g+dGpF4ydZeehkRYYMa08vCXamW2iwDhR6YCKMdJ7noQxkQXvJI64RmZo3WKPa071TkvMuQ== X-Received: by 2002:a05:600c:4e52:b0:3b4:91ec:b15d with SMTP id e18-20020a05600c4e5200b003b491ecb15dmr9082888wmq.119.1666266914938; Thu, 20 Oct 2022 04:55:14 -0700 (PDT) Received: from p19180.home ([2a01:cb1c:e35:9600:4170:2ec:443b:608d]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c190800b003a601a1c2f7sm3005291wmq.19.2022.10.20.04.55.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Oct 2022 04:55:14 -0700 (PDT) From: Victor Dumas To: buildroot@buildroot.org Date: Thu, 20 Oct 2022 13:55:12 +0200 Message-Id: <20221020115512.483686-1-dumasv.dev@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221019124017.427199-1-dumasv.dev@gmail.com> References: <20221019124017.427199-1-dumasv.dev@gmail.com> MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nl6gBuDkdBBhlJUsdoEyoRtZB7CmnWhJWX2Ky56AqZ4=; b=d9Cfi71xWkk5zzHM04QYrx7CSoUdl0LfuFWUSefSmyYI/t7Kntg05feUmAWiScvMrC VPoiNyO+bbYqDuhGLemnzYOlqka5hfF4cx/S/URl5ZLPLzK/EJ8PAy7UpRdO93cLJgQd I8DSyF0riuUdyiZedwGJWTkpS3jctk+4SW+IMKzI0xd5rQYjcuKF+pO6xkQWELyBO4WM 9sUtwDi7lRkAQT63iMn4Y2rgpYIR3gzkHSWgWVArfct/TvFOxZaUTi0j3fjrC8Go27aL bcWKhSo960OKYGhBCXjQsgia41WWtjaG1NF69RCJILL9xRkQum/7OBRdXx4iD8BnmZB3 a2CA== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=d9Cfi71x Subject: [Buildroot] [PATCH v4] support/scripts/fix-rpath: parallelize patching files X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Victor Dumas Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" Using "xargs" instead of "while read" loop allows for the patching of files to be parallelized This significantly reduces the amount of time it takes to fix all the paths. On a larger RFS(~300MB) this script was taking 5 minutes, it now only takes about 30s on a 12 core machine Signed-off-by: Victor Dumas --- support/scripts/fix-rpath | 66 ++++++++++++++++++++++----------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath index 3e67e770e5..cef7a48e76 100755 --- a/support/scripts/fix-rpath +++ b/support/scripts/fix-rpath @@ -58,6 +58,38 @@ HOST_EXCLUDEPATHS="/share/terminfo" STAGING_EXCLUDEPATHS="/usr/include /usr/share/terminfo" TARGET_EXCLUDEPATHS="/lib/firmware" +patch_file() { + PATCHELF="${1}" + rootdir="${2}" + sanitize_extra_args="${3}" + file="${4}" + + # check if it's an ELF file + rpath=$(${PATCHELF} --print-rpath "${file}" 2>&1) + if test $? -ne 0 ; then + return 0 + fi + + # make files writable if necessary + changed=$(chmod -c u+w "${file}") + + # With per-package directory support, most RPATH of host + # binaries will point to per-package directories. This won't + # work with the --make-rpath-relative ${rootdir} invocation as + # the per-package host directory is not within ${rootdir}. So, + # we rewrite all RPATHs pointing to per-package directories so + # that they point to the global host directry. + changed_rpath=$(echo ${rpath} | sed "s@${PER_PACKAGE_DIR}/[^/]\+/host@${HOST_DIR}@") + if test "${rpath}" != "${changed_rpath}" ; then + ${PATCHELF} --set-rpath ${changed_rpath} "${file}" + fi + + # call patchelf to sanitize the rpath + ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}" + # restore the original permission + test "${changed}" != "" && chmod u-w "${file}" +} + main() { local rootdir local tree="${1}" @@ -123,34 +155,12 @@ main() { ;; esac - find_args+=( "-type" "f" "-print" ) - - while read file ; do - # check if it's an ELF file - rpath=$(${PATCHELF} --print-rpath "${file}" 2>&1) - if test $? -ne 0 ; then - continue - fi - - # make files writable if necessary - changed=$(chmod -c u+w "${file}") - - # With per-package directory support, most RPATH of host - # binaries will point to per-package directories. This won't - # work with the --make-rpath-relative ${rootdir} invocation as - # the per-package host directory is not within ${rootdir}. So, - # we rewrite all RPATHs pointing to per-package directories so - # that they point to the global host directry. - changed_rpath=$(echo ${rpath} | sed "s@${PER_PACKAGE_DIR}/[^/]\+/host@${HOST_DIR}@") - if test "${rpath}" != "${changed_rpath}" ; then - ${PATCHELF} --set-rpath ${changed_rpath} "${file}" - fi - - # call patchelf to sanitize the rpath - ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}" - # restore the original permission - test "${changed}" != "" && chmod u-w "${file}" - done < <(find "${rootdir}" ${find_args[@]}) + find_args+=( "-type" "f" "-print0" ) + + export -f patch_file + # Limit the number of cores used + procs=$(echo -e "$(($(nproc)-2)) \n 1" | sort -n | tail -n1) + find "${rootdir}" ${find_args[@]} | xargs -0 -r -P ${procs} -I {} bash -c "patch_file '${PATCHELF}' '${rootdir}' '${sanitize_extra_args}' $@" _ {} # Restore patched patchelf utility test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}" -- 2.30.2 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot