* [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys
@ 2025-03-18 23:24 Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
` (4 more replies)
0 siblings, 5 replies; 21+ messages in thread
From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw)
To: git; +Cc: brian m. carlson
Fix a variety of issues with `cargo package` for the libgit-sys crate.
Most of these are straightforward, but patch #2 carries an unfortunate
side-effect that building the libgit-sys crate requires cleaning the Git
worktree in the process. Please see the patch description there for
discussion of possible alternatives, and please suggest any other ideas
you might have for that issue.
`cargo package` for the libgit-rs crate is still broken for now, because
it will require publishing the the libgit-sys crate first.
Josh Steadmon (4):
libgitpub: move to separate contrib/ directory
libgit-sys: add symlink to git repo root and clean after build
libgit-sys: parallelize build with Cargo's jobserver
libgit-sys: exclude unnecessary directories in git-src
Makefile | 36 ++++++++++++++-----
contrib/libgit-sys/Cargo.toml | 12 +++++++
contrib/libgit-sys/build.rs | 34 ++++++++++++++++--
contrib/libgit-sys/git-src | 1 +
.../public_symbol_export.c | 2 +-
.../public_symbol_export.h | 0
6 files changed, 72 insertions(+), 13 deletions(-)
create mode 120000 contrib/libgit-sys/git-src
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%)
rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%)
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
--
2.49.0.rc1.451.g8f38331e32-goog
^ permalink raw reply [flat|nested] 21+ messages in thread* [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory 2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon @ 2025-03-18 23:24 ` Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon ` (3 subsequent siblings) 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw) To: git; +Cc: brian m. carlson For Cargo packaging purposes, it is better for any dependencies built with Git's Makefile to live outside the libgit-{sys,rs} crates. This, plus some future changes to the crates, will allow `cargo package` to treat all of the non-Rust source as a separate tree apart from the Cargo-managed inputs. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 16 ++++++++-------- contrib/libgit-sys/build.rs | 4 ++-- .../public_symbol_export.c | 2 +- .../public_symbol_export.h | 0 4 files changed, 11 insertions(+), 11 deletions(-) rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%) rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%) diff --git a/Makefile b/Makefile index 7315507381..52eed88dde 100644 --- a/Makefile +++ b/Makefile @@ -2764,7 +2764,7 @@ OBJECTS += $(CLAR_TEST_OBJS) OBJECTS += $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS)) ifdef INCLUDE_LIBGIT_RS - OBJECTS += contrib/libgit-sys/public_symbol_export.o + OBJECTS += contrib/libgitpub/public_symbol_export.o endif ifndef NO_CURL @@ -3762,9 +3762,9 @@ clean: profile-clean coverage-clean cocciclean $(MAKE) -C Documentation/ clean $(RM) Documentation/GIT-EXCLUDED-PROGRAMS $(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target - $(RM) contrib/libgit-sys/partial_symbol_export.o - $(RM) contrib/libgit-sys/hidden_symbol_export.o - $(RM) contrib/libgit-sys/libgitpub.a + $(RM) contrib/libgitpub/partial_symbol_export.o + $(RM) contrib/libgitpub/hidden_symbol_export.o + $(RM) contrib/libgitpub/libgitpub.a ifndef NO_PERL $(RM) -r perl/build/ endif @@ -3937,14 +3937,14 @@ ifdef INCLUDE_LIBGIT_RS all:: libgit-sys libgit-rs endif -LIBGIT_PUB_OBJS += contrib/libgit-sys/public_symbol_export.o +LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o LIBGIT_PUB_OBJS += libgit.a LIBGIT_PUB_OBJS += reftable/libreftable.a LIBGIT_PUB_OBJS += xdiff/lib.a -LIBGIT_PARTIAL_EXPORT = contrib/libgit-sys/partial_symbol_export.o +LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o -LIBGIT_HIDDEN_EXPORT = contrib/libgit-sys/hidden_symbol_export.o +LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LD) -r $^ -o $@ @@ -3952,5 +3952,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT) $(OBJCOPY) --localize-hidden $^ $@ -contrib/libgit-sys/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT) +contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT) $(AR) $(ARFLAGS) $@ $^ diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 3ffd80ad91..e0d979c196 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -15,7 +15,7 @@ pub fn main() -> std::io::Result<()> { .current_dir(git_root.clone()) .args([ "INCLUDE_LIBGIT_RS=YesPlease", - "contrib/libgit-sys/libgitpub.a", + "contrib/libgitpub/libgitpub.a", ]) .output() .expect("Make failed to run"); @@ -26,7 +26,7 @@ pub fn main() -> std::io::Result<()> { String::from_utf8(make_output.stderr).unwrap() ); } - std::fs::copy(crate_root.join("libgitpub.a"), dst.join("libgitpub.a"))?; + std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?; println!("cargo:rustc-link-search=native={}", dst.display()); println!("cargo:rustc-link-lib=gitpub"); println!("cargo:rerun-if-changed={}", git_root.display()); diff --git a/contrib/libgit-sys/public_symbol_export.c b/contrib/libgitpub/public_symbol_export.c similarity index 96% rename from contrib/libgit-sys/public_symbol_export.c rename to contrib/libgitpub/public_symbol_export.c index dfbb257115..de5f6943c2 100644 --- a/contrib/libgit-sys/public_symbol_export.c +++ b/contrib/libgitpub/public_symbol_export.c @@ -6,7 +6,7 @@ #include "git-compat-util.h" #include "config.h" -#include "contrib/libgit-sys/public_symbol_export.h" +#include "contrib/libgitpub/public_symbol_export.h" #include "version.h" #pragma GCC visibility push(default) diff --git a/contrib/libgit-sys/public_symbol_export.h b/contrib/libgitpub/public_symbol_export.h similarity index 100% rename from contrib/libgit-sys/public_symbol_export.h rename to contrib/libgitpub/public_symbol_export.h -- 2.49.0.rc1.451.g8f38331e32-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build 2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon @ 2025-03-18 23:24 ` Josh Steadmon 2025-03-19 22:23 ` Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon ` (2 subsequent siblings) 4 siblings, 1 reply; 21+ messages in thread From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw) To: git; +Cc: brian m. carlson Unlike `cargo build`, `cargo package` does not get access to the entire Git repo containing a Rust crate. Instead, it prepares a directory starting from the crate root (potentially excluding files, such as those not under version control, or explicity excluded in the Cargo.toml file). This means that the current method of building the libgit-sys crate does not work with `cargo package`, as it tries to execute the Makefile from "../.." relative to the crate root. Fix this by adding a `git-src` symlink in the crate that points to the Git repository root. `cargo package` will flatten this to a copy of the Git repo, excluding non-version-controlled files, any explicitly-excluded files, and trees that contain a Cargo.toml file (this prevents infinite recursion on the symlink). We can then execute the Makefile under the flattened git-src directory from our build.rs script. However, this exposes a second problem; Cargo will check that the build script does not add, delete, or modify any source files. This means that after we copy our libgitpub.a dependency to the output directory, we must run `make clean` to remove the object files we created during the build process. Unfortunately, there is not a way to determine from the build.rs script whether we're running `cargo build` vs. `cargo package`, so now any build of the libgit-sys crate will result in cleaning the Git worktree. A potential alternative is to make an additional temporary copy of the worktree and run the Makefile there. This would avoid removing build artifacts in the worktree at the cost of copying MBs worth of source files to a temporary directory. Perhaps hardlinking instead of making a full copy would help here, but that might be less portable. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 6 ++++++ contrib/libgit-sys/build.rs | 21 ++++++++++++++++++++- contrib/libgit-sys/git-src | 1 + 3 files changed, 27 insertions(+), 1 deletion(-) create mode 120000 contrib/libgit-sys/git-src diff --git a/Makefile b/Makefile index 52eed88dde..e7d8786e4e 100644 --- a/Makefile +++ b/Makefile @@ -420,6 +420,10 @@ include shared.mak # Define INCLUDE_LIBGIT_RS if you want `make all` and `make test` to build and # test the Rust crates in contrib/libgit-sys and contrib/libgit-rs. # +# Define PRESERVE_LIBGIT_TARGET to prevent `make clean` from removing the Cargo +# output directories for libgit-sys and libgit-rs. This is mainly for use in the +# Cargo build scripts. +# # === Optional library: libintl === # # Define NO_GETTEXT if you don't want Git output to be translated. @@ -3761,7 +3765,9 @@ clean: profile-clean coverage-clean cocciclean $(RM) $(htmldocs).tar.gz $(manpages).tar.gz $(MAKE) -C Documentation/ clean $(RM) Documentation/GIT-EXCLUDED-PROGRAMS +ifndef PRESERVE_LIBGIT_TARGET $(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target +endif $(RM) contrib/libgitpub/partial_symbol_export.o $(RM) contrib/libgitpub/hidden_symbol_export.o $(RM) contrib/libgitpub/libgitpub.a diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index e0d979c196..9d586d272d 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -6,7 +6,7 @@ pub fn main() -> std::io::Result<()> { ac.emit_has_path("std::ffi::c_char"); let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); - let git_root = crate_root.join("../.."); + let git_root = crate_root.join("git-src"); let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap()); let make_output = make_cmd::gnu_make() @@ -31,5 +31,24 @@ pub fn main() -> std::io::Result<()> { println!("cargo:rustc-link-lib=gitpub"); println!("cargo:rerun-if-changed={}", git_root.display()); + let make_output = make_cmd::gnu_make() + .env("DEVELOPER", "1") + .env_remove("PROFILE") + .current_dir(git_root.clone()) + .args([ + "INCLUDE_LIBGIT_RS=YesPlease", + "PRESERVE_LIBGIT_TARGET=YesPlease", + "clean", + ]) + .output() + .expect("`make clean` failed to run"); + if !make_output.status.success() { + panic!( + "`make clean` failed:\n stdout = {}\n stderr = {}\n", + String::from_utf8(make_output.stdout).unwrap(), + String::from_utf8(make_output.stderr).unwrap() + ); + } + Ok(()) } diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src new file mode 120000 index 0000000000..c25bddb6dd --- /dev/null +++ b/contrib/libgit-sys/git-src @@ -0,0 +1 @@ +../.. \ No newline at end of file -- 2.49.0.rc1.451.g8f38331e32-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build 2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon @ 2025-03-19 22:23 ` Josh Steadmon 2025-03-20 11:10 ` Phillip Wood 0 siblings, 1 reply; 21+ messages in thread From: Josh Steadmon @ 2025-03-19 22:23 UTC (permalink / raw) To: git; +Cc: brian m. carlson On 2025.03.18 16:24, Josh Steadmon wrote: > Unlike `cargo build`, `cargo package` does not get access to the entire Git repo > containing a Rust crate. Instead, it prepares a directory starting from the > crate root (potentially excluding files, such as those not under version > control, or explicity excluded in the Cargo.toml file). > > This means that the current method of building the libgit-sys crate does not > work with `cargo package`, as it tries to execute the Makefile from "../.." > relative to the crate root. > > Fix this by adding a `git-src` symlink in the crate that points to the Git > repository root. `cargo package` will flatten this to a copy of the Git repo, > excluding non-version-controlled files, any explicitly-excluded files, and trees > that contain a Cargo.toml file (this prevents infinite recursion on the > symlink). > > We can then execute the Makefile under the flattened git-src directory from our > build.rs script. However, this exposes a second problem; Cargo will check that > the build script does not add, delete, or modify any source files. This means > that after we copy our libgitpub.a dependency to the output directory, we must > run `make clean` to remove the object files we created during the build process. > > Unfortunately, there is not a way to determine from the build.rs script whether > we're running `cargo build` vs. `cargo package`, so now any build of the > libgit-sys crate will result in cleaning the Git worktree. > > A potential alternative is to make an additional temporary copy of the worktree > and run the Makefile there. This would avoid removing build artifacts in the > worktree at the cost of copying MBs worth of source files to a temporary > directory. Perhaps hardlinking instead of making a full copy would help here, > but that might be less portable. I'm currently working on an alternate solution where we build the object files in Cargo's working directory, rather than in the source tree. This will (mostly) avoid the need to clean after the build. I hope to send a v2 with this change in the next few days. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build 2025-03-19 22:23 ` Josh Steadmon @ 2025-03-20 11:10 ` Phillip Wood 2025-03-21 19:49 ` Josh Steadmon 0 siblings, 1 reply; 21+ messages in thread From: Phillip Wood @ 2025-03-20 11:10 UTC (permalink / raw) To: Josh Steadmon, git, brian m. carlson Hi Josh On 19/03/2025 22:23, Josh Steadmon wrote: > On 2025.03.18 16:24, Josh Steadmon wrote: > > I'm currently working on an alternate solution where we build the object > files in Cargo's working directory, rather than in the source tree. This > will (mostly) avoid the need to clean after the build. I hope to send a > v2 with this change in the next few days. Meson has builtin support for out-of-tree builds which would make building in Cargo's OUT_DIR trivial. Our meson build is still experimental though so you may not want to rely on it. Best Wishes Phillip ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build 2025-03-20 11:10 ` Phillip Wood @ 2025-03-21 19:49 ` Josh Steadmon 0 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 19:49 UTC (permalink / raw) To: phillip.wood; +Cc: git, brian m. carlson On 2025.03.20 11:10, Phillip Wood wrote: > Hi Josh > > On 19/03/2025 22:23, Josh Steadmon wrote: > > On 2025.03.18 16:24, Josh Steadmon wrote: > > > > I'm currently working on an alternate solution where we build the object > > files in Cargo's working directory, rather than in the source tree. This > > will (mostly) avoid the need to clean after the build. I hope to send a > > v2 with this change in the next few days. > > Meson has builtin support for out-of-tree builds which would make building > in Cargo's OUT_DIR trivial. Our meson build is still experimental though so > you may not want to rely on it. > > Best Wishes > > Phillip Thanks for the pointer! I have a working Makefile solution which I'll be sending out soon, but I'll keep meson in mind as a backup if the list doesn't like my V2. ^ permalink raw reply [flat|nested] 21+ messages in thread
* [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver 2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon @ 2025-03-18 23:24 ` Josh Steadmon 2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw) To: git; +Cc: brian m. carlson Cargo provides GNU Make flags that can be used to connect to its jobserver for managing parallel builds. They are not automatically passed to any `make` invocations, so let's add that to our build.rs script. Signed-off-by: Josh Steadmon <steadmon@google.com> --- contrib/libgit-sys/build.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 9d586d272d..16e0d66afb 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -8,9 +8,11 @@ pub fn main() -> std::io::Result<()> { let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); let git_root = crate_root.join("git-src"); let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap()); + let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap(); let make_output = make_cmd::gnu_make() .env("DEVELOPER", "1") + .env("MAKEFLAGS", &makeflags) .env_remove("PROFILE") .current_dir(git_root.clone()) .args([ @@ -33,6 +35,7 @@ pub fn main() -> std::io::Result<()> { let make_output = make_cmd::gnu_make() .env("DEVELOPER", "1") + .env("MAKEFLAGS", &makeflags) .env_remove("PROFILE") .current_dir(git_root.clone()) .args([ -- 2.49.0.rc1.451.g8f38331e32-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src 2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon ` (2 preceding siblings ...) 2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon @ 2025-03-18 23:24 ` Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-18 23:24 UTC (permalink / raw) To: git; +Cc: brian m. carlson We can avoid copying tens of megabytes of unnecessary source files by excluding a few directories which are not needed to compile libgitpub.a. This helps us stay below crates.io's 10 MB size limit. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 14 +++++++++++++- contrib/libgit-sys/Cargo.toml | 12 ++++++++++++ contrib/libgit-sys/build.rs | 8 +++++++- 3 files changed, 32 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index e7d8786e4e..c75f1d0208 100644 --- a/Makefile +++ b/Makefile @@ -651,6 +651,7 @@ export prefix bindir sharedir sysconfdir perllibdir localedir CC = cc AR = ar RM = rm -f +RMDIR = rmdir --ignore-fail-on-non-empty DIFF = diff TAR = tar FIND = find @@ -3477,13 +3478,13 @@ coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE) # "Sub"-Makefiles, not really because they can't be run stand-alone, # only there to contain directory-specific rules and variables +ifndef NO_GITWEB ## gitweb/Makefile inclusion: MAK_DIR_GITWEB = gitweb/ include gitweb/Makefile .PHONY: gitweb gitweb: $(MAK_DIR_GITWEB_ALL) -ifndef NO_GITWEB all:: gitweb endif @@ -3763,7 +3764,9 @@ clean: profile-clean coverage-clean cocciclean $(RM) -r .dist-tmp-dir .doc-tmp-dir $(RM) $(GIT_TARNAME).tar.gz $(RM) $(htmldocs).tar.gz $(manpages).tar.gz +ifndef INCLUDE_LIBGIT_RS $(MAKE) -C Documentation/ clean +endif $(RM) Documentation/GIT-EXCLUDED-PROGRAMS ifndef PRESERVE_LIBGIT_TARGET $(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target @@ -3775,7 +3778,9 @@ ifndef NO_PERL $(RM) -r perl/build/ endif $(MAKE) -C templates/ clean +ifndef INCLUDE_LIBGIT_RS $(MAKE) -C t/ clean +endif ifndef NO_TCLTK $(MAKE) -C gitk-git clean $(MAKE) -C git-gui clean @@ -3798,6 +3803,13 @@ ifdef MSVC $(RM) compat/vcbuild/MSVC-DEFS-GEN endif +# Handle additional cleanup needed for running `cargo package` for libgit-sys. +ifdef INCLUDE_LIBGIT_RS +libgit-pkg-clean: clean + $(RMDIR) t/helper t/unit-tests/clar t/unit-tests t + $(RMDIR) oss-fuzz +endif + .PHONY: all install profile-clean cocciclean clean strip .PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell .PHONY: FORCE diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml index e0623022c3..4b8facccfe 100644 --- a/contrib/libgit-sys/Cargo.toml +++ b/contrib/libgit-sys/Cargo.toml @@ -7,6 +7,18 @@ links = "gitpub" rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from # version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/ description = "Native bindings to a portion of libgit" +exclude = [ + "git-src/.github", + "git-src/ci", + "git-src/Documentation", + "git-src/git-gui", + "git-src/gitk-git", + "git-src/gitweb", + "git-src/oss-fuzz", + "git-src/perl", + "git-src/po", + "git-src/t" +] [lib] path = "src/lib.rs" diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 16e0d66afb..0a72040726 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -17,6 +17,9 @@ pub fn main() -> std::io::Result<()> { .current_dir(git_root.clone()) .args([ "INCLUDE_LIBGIT_RS=YesPlease", + "NO_GITWEB=YesPlease", + "NO_PERL=YesPlease", + "NO_TCLTK=YesPlease", "contrib/libgitpub/libgitpub.a", ]) .output() @@ -40,8 +43,11 @@ pub fn main() -> std::io::Result<()> { .current_dir(git_root.clone()) .args([ "INCLUDE_LIBGIT_RS=YesPlease", + "NO_GITWEB=YesPlease", + "NO_PERL=YesPlease", + "NO_TCLTK=YesPlease", "PRESERVE_LIBGIT_TARGET=YesPlease", - "clean", + "libgit-pkg-clean", ]) .output() .expect("`make clean` failed to run"); -- 2.49.0.rc1.451.g8f38331e32-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 0/5] Fix `cargo package` for libgit-sys 2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon ` (3 preceding siblings ...) 2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon ` (4 more replies) 4 siblings, 5 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood Fix `cargo package` for the libgit-sys crate by providing a pristine copy of the Git source tree at package time, and by building object files and other generated files outside of this tree. While we're at it, improve parallel builds by plumbing Cargo's make flags to the make command invoked in our build.rs script. We also fix a few problems that would prevent us from publishing the libgit-sys crate: we add a license and description to Cargo.toml, and we exclude unnecessary files from the packaged source, to stay below crates.io's 10 MB limit. `cargo package` for the libgit-rs crate is still broken for now, because it will require publishing the libgit-sys crate first. Changes from V1: * Reworked patch #2 to build outside of the source tree rather than running `make clean` after the build. * Simplified patch #4 now that cleaning is no longer necessary. * Added patch #5 to add some required Cargo.toml fields. Josh Steadmon (5): libgitpub: move to separate contrib/ directory libgit-sys: add symlink to git repo root and build out of tree libgit-sys: parallelize build with Cargo's jobserver libgit-sys: exclude unnecessary directories in git-src libgit-{sys,rs}: add license and description fields Makefile | 145 +++++++++++------- contrib/libgit-rs/Cargo.toml | 2 + contrib/libgit-sys/Cargo.toml | 13 ++ contrib/libgit-sys/build.rs | 12 +- contrib/libgit-sys/git-src | 1 + .../public_symbol_export.c | 2 +- .../public_symbol_export.h | 0 shared.mak | 5 + 8 files changed, 117 insertions(+), 63 deletions(-) create mode 120000 contrib/libgit-sys/git-src rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%) rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%) Range-diff against v1: 1: 28e10e1092 = 1: 28e10e1092 libgitpub: move to separate contrib/ directory 2: 58111043a1 < -: ---------- libgit-sys: add symlink to git repo root and clean after build -: ---------- > 2: 6befc95a2d libgit-sys: add symlink to git repo root and build out of tree 3: 4cf9996096 ! 3: a34e23a83d libgit-sys: parallelize build with Cargo's jobserver @@ contrib/libgit-sys/build.rs: pub fn main() -> std::io::Result<()> { let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap()); + let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap(); - let make_output = make_cmd::gnu_make() - .env("DEVELOPER", "1") -+ .env("MAKEFLAGS", &makeflags) - .env_remove("PROFILE") - .current_dir(git_root.clone()) - .args([ -@@ contrib/libgit-sys/build.rs: pub fn main() -> std::io::Result<()> { - let make_output = make_cmd::gnu_make() .env("DEVELOPER", "1") + .env("MAKEFLAGS", &makeflags) 4: 9b5d4aa140 < -: ---------- libgit-sys: exclude unnecessary directories in git-src -: ---------- > 4: 390695ac1f libgit-sys: exclude unnecessary directories in git-src -: ---------- > 5: 3a87f54693 libgit-{sys,rs}: add license and description fields base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v2 1/5] libgitpub: move to separate contrib/ directory 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon ` (3 subsequent siblings) 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood For Cargo packaging purposes, it is better for any dependencies built with Git's Makefile to live outside the libgit-{sys,rs} crates. This, plus some future changes to the crates, will allow `cargo package` to treat all of the non-Rust source as a separate tree apart from the Cargo-managed inputs. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 16 ++++++++-------- contrib/libgit-sys/build.rs | 4 ++-- .../public_symbol_export.c | 2 +- .../public_symbol_export.h | 0 4 files changed, 11 insertions(+), 11 deletions(-) rename contrib/{libgit-sys => libgitpub}/public_symbol_export.c (96%) rename contrib/{libgit-sys => libgitpub}/public_symbol_export.h (100%) diff --git a/Makefile b/Makefile index 7315507381..52eed88dde 100644 --- a/Makefile +++ b/Makefile @@ -2764,7 +2764,7 @@ OBJECTS += $(CLAR_TEST_OBJS) OBJECTS += $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TEST_PROGRAMS)) ifdef INCLUDE_LIBGIT_RS - OBJECTS += contrib/libgit-sys/public_symbol_export.o + OBJECTS += contrib/libgitpub/public_symbol_export.o endif ifndef NO_CURL @@ -3762,9 +3762,9 @@ clean: profile-clean coverage-clean cocciclean $(MAKE) -C Documentation/ clean $(RM) Documentation/GIT-EXCLUDED-PROGRAMS $(RM) -r contrib/libgit-sys/target contrib/libgit-rs/target - $(RM) contrib/libgit-sys/partial_symbol_export.o - $(RM) contrib/libgit-sys/hidden_symbol_export.o - $(RM) contrib/libgit-sys/libgitpub.a + $(RM) contrib/libgitpub/partial_symbol_export.o + $(RM) contrib/libgitpub/hidden_symbol_export.o + $(RM) contrib/libgitpub/libgitpub.a ifndef NO_PERL $(RM) -r perl/build/ endif @@ -3937,14 +3937,14 @@ ifdef INCLUDE_LIBGIT_RS all:: libgit-sys libgit-rs endif -LIBGIT_PUB_OBJS += contrib/libgit-sys/public_symbol_export.o +LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o LIBGIT_PUB_OBJS += libgit.a LIBGIT_PUB_OBJS += reftable/libreftable.a LIBGIT_PUB_OBJS += xdiff/lib.a -LIBGIT_PARTIAL_EXPORT = contrib/libgit-sys/partial_symbol_export.o +LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o -LIBGIT_HIDDEN_EXPORT = contrib/libgit-sys/hidden_symbol_export.o +LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LD) -r $^ -o $@ @@ -3952,5 +3952,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT) $(OBJCOPY) --localize-hidden $^ $@ -contrib/libgit-sys/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT) +contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT) $(AR) $(ARFLAGS) $@ $^ diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 3ffd80ad91..e0d979c196 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -15,7 +15,7 @@ pub fn main() -> std::io::Result<()> { .current_dir(git_root.clone()) .args([ "INCLUDE_LIBGIT_RS=YesPlease", - "contrib/libgit-sys/libgitpub.a", + "contrib/libgitpub/libgitpub.a", ]) .output() .expect("Make failed to run"); @@ -26,7 +26,7 @@ pub fn main() -> std::io::Result<()> { String::from_utf8(make_output.stderr).unwrap() ); } - std::fs::copy(crate_root.join("libgitpub.a"), dst.join("libgitpub.a"))?; + std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?; println!("cargo:rustc-link-search=native={}", dst.display()); println!("cargo:rustc-link-lib=gitpub"); println!("cargo:rerun-if-changed={}", git_root.display()); diff --git a/contrib/libgit-sys/public_symbol_export.c b/contrib/libgitpub/public_symbol_export.c similarity index 96% rename from contrib/libgit-sys/public_symbol_export.c rename to contrib/libgitpub/public_symbol_export.c index dfbb257115..de5f6943c2 100644 --- a/contrib/libgit-sys/public_symbol_export.c +++ b/contrib/libgitpub/public_symbol_export.c @@ -6,7 +6,7 @@ #include "git-compat-util.h" #include "config.h" -#include "contrib/libgit-sys/public_symbol_export.h" +#include "contrib/libgitpub/public_symbol_export.h" #include "version.h" #pragma GCC visibility push(default) diff --git a/contrib/libgit-sys/public_symbol_export.h b/contrib/libgitpub/public_symbol_export.h similarity index 100% rename from contrib/libgit-sys/public_symbol_export.h rename to contrib/libgitpub/public_symbol_export.h -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 2025-03-23 1:46 ` Eric Sunshine 2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon ` (2 subsequent siblings) 4 siblings, 1 reply; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood Unlike `cargo build`, `cargo package` does not get access to the entire Git repo containing a Rust crate. Instead, it prepares a directory starting from the crate root (potentially excluding files, such as those not under version control, or explicity excluded in the Cargo.toml file). This means that the current method of building the libgit-sys crate does not work with `cargo package`, as it tries to execute the Makefile from "../.." relative to the crate root. Fix this by adding a `git-src` symlink in the crate that points to the Git repository root. `cargo package` will flatten this to a copy of the Git repo, excluding non-version-controlled files, any explicitly-excluded files, and trees that contain a Cargo.toml file (this prevents infinite recursion on the symlink). We can then execute the Makefile under the flattened git-src directory from our build.rs script. However, this exposes a second problem; Cargo will check that the build script does not add, delete, or modify any files in the source tree. Without further changes, Cargo complains about the object files and other generated files created during the build. To avoid this problem, add a CARGO_OUT_DIR variable to the Makefile. When this is set, object files and other generated files will be created there, rather than in the main source tree. This change has only been applied to files created as part of the libgitpub.a build, to avoid unnecessary churn in the Makefile. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 135 +++++++++++++++++++++--------------- contrib/libgit-sys/build.rs | 7 +- contrib/libgit-sys/git-src | 1 + shared.mak | 5 ++ 4 files changed, 91 insertions(+), 57 deletions(-) create mode 120000 contrib/libgit-sys/git-src diff --git a/Makefile b/Makefile index 52eed88dde..91677448ba 100644 --- a/Makefile +++ b/Makefile @@ -420,6 +420,9 @@ include shared.mak # Define INCLUDE_LIBGIT_RS if you want `make all` and `make test` to build and # test the Rust crates in contrib/libgit-sys and contrib/libgit-rs. # +# Define CARGO_OUT_DIR to specify a directory where object files and other files +# generated during the build of libgitpub.a should be created. +# # === Optional library: libintl === # # Define NO_GETTEXT if you don't want Git output to be translated. @@ -699,6 +702,7 @@ THIRD_PARTY_SOURCES = UNIT_TEST_PROGRAMS = UNIT_TEST_DIR = t/unit-tests UNIT_TEST_BIN = $(UNIT_TEST_DIR)/bin +CARGO_OUT_DIR = # Having this variable in your environment would break pipelines because # you cause "cd" to echo its destination to stdout. It can also take @@ -917,11 +921,21 @@ export PYTHON_PATH TEST_SHELL_PATH = $(SHELL_PATH) -LIB_FILE = libgit.a -XDIFF_LIB = xdiff/lib.a -REFTABLE_LIB = reftable/libreftable.a +### Generated files which may need to live in Cargo output directory + +GIT_CFLAGS = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-CFLAGS) +GIT_PREFIX = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-PREFIX) +GIT_USER_AGENT_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-USER-AGENT) +GIT_VERSION_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),GIT-VERSION-FILE) +GIT_VERSION_PATH = $(dir $(GIT_VERSION_FILE))$(notdir $(GIT_VERSION_FILE)) +COMMAND_LIST_H = $(call maybe_join_path,$(CARGO_OUT_DIR),command-list.h) +VERSION_DEF_H = $(call maybe_join_path,$(CARGO_OUT_DIR),version-def.h) -GENERATED_H += command-list.h +LIB_FILE = $(call maybe_join_path,$(CARGO_OUT_DIR),libgit.a) +XDIFF_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),xdiff/lib.a) +REFTABLE_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),reftable/libreftable.a) + +GENERATED_H += $(COMMAND_LIST_H) GENERATED_H += config-list.h GENERATED_H += hook-list.h GENERATED_H += $(UNIT_TEST_DIR)/clar-decls.h @@ -1472,7 +1486,7 @@ ifdef DEVELOPER include config.mak.dev endif -GIT-VERSION-FILE: FORCE +$(GIT_VERSION_FILE): FORCE @OLD=$$(cat $@ 2>/dev/null || :) && \ $(call version_gen,"$(shell pwd)",GIT-VERSION-FILE.in,$@) && \ NEW=$$(cat $@ 2>/dev/null || :) && \ @@ -1482,7 +1496,7 @@ GIT-VERSION-FILE: FORCE # otherwise any user-provided value for GIT_VERSION would have been overridden # already. GIT_VERSION_OVERRIDE := $(GIT_VERSION) --include GIT-VERSION-FILE +-include $(GIT_VERSION_FILE) # what 'all' will build and 'install' will install in gitexecdir, # excluding programs for built-in commands @@ -2403,9 +2417,9 @@ endif GIT_USER_AGENT_SQ = $(subst ','\'',$(GIT_USER_AGENT)) GIT_USER_AGENT_CQ = "$(subst ",\",$(subst \,\\,$(GIT_USER_AGENT)))" GIT_USER_AGENT_CQ_SQ = $(subst ','\'',$(GIT_USER_AGENT_CQ)) -GIT-USER-AGENT: FORCE - @if test x'$(GIT_USER_AGENT_SQ)' != x"`cat GIT-USER-AGENT 2>/dev/null`"; then \ - echo '$(GIT_USER_AGENT_SQ)' >GIT-USER-AGENT; \ +$(GIT_USER_AGENT_FILE): FORCE + @if test x'$(GIT_USER_AGENT_SQ)' != x"`cat $(GIT_USER_AGENT_FILE) 2>/dev/null`"; then \ + echo '$(GIT_USER_AGENT_SQ)' >$(GIT_USER_AGENT_FILE); \ fi ifdef DEFAULT_HELP_FORMAT @@ -2523,7 +2537,7 @@ strip: $(PROGRAMS) git$X # dependencies here will not need to change if the force-build # details change some day. -git.sp git.s git.o: GIT-PREFIX +git.sp git.s git.o: $(GIT_PREFIX) git.sp git.s git.o: EXTRA_CPPFLAGS = \ '-DGIT_HTML_PATH="$(htmldir_relative_SQ)"' \ '-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \ @@ -2533,10 +2547,10 @@ git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ $(filter %.o,$^) $(LIBS) -help.sp help.s help.o: command-list.h +help.sp help.s $(call maybe_join_path,$(CARGO_OUT_DIR),help.o): $(COMMAND_LIST_H) builtin/bugreport.sp builtin/bugreport.s builtin/bugreport.o: hook-list.h -builtin/help.sp builtin/help.s builtin/help.o: config-list.h GIT-PREFIX +builtin/help.sp builtin/help.s builtin/help.o: config-list.h $(GIT_PREFIX) builtin/help.sp builtin/help.s builtin/help.o: EXTRA_CPPFLAGS = \ '-DGIT_HTML_PATH="$(htmldir_relative_SQ)"' \ '-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \ @@ -2545,13 +2559,13 @@ builtin/help.sp builtin/help.s builtin/help.o: EXTRA_CPPFLAGS = \ PAGER_ENV_SQ = $(subst ','\'',$(PAGER_ENV)) PAGER_ENV_CQ = "$(subst ",\",$(subst \,\\,$(PAGER_ENV)))" PAGER_ENV_CQ_SQ = $(subst ','\'',$(PAGER_ENV_CQ)) -pager.sp pager.s pager.o: EXTRA_CPPFLAGS = \ +pager.sp pager.s $(call maybe_join_path,$(CARGO_OUT_DIR),pager.o): EXTRA_CPPFLAGS = \ -DPAGER_ENV='$(PAGER_ENV_CQ_SQ)' -version-def.h: version-def.h.in GIT-VERSION-GEN GIT-VERSION-FILE GIT-USER-AGENT +$(VERSION_DEF_H): version-def.h.in GIT-VERSION-GEN $(GIT_VERSION_FILE) $(GIT_USER_AGENT_FILE) $(QUIET_GEN)$(call version_gen,"$(shell pwd)",$<,$@) -version.sp version.s version.o: version-def.h +version.sp version.s $(call maybe_join_path,$(CARGO_OUT_DIR),version.o): $(VERSION_DEF_H) $(BUILT_INS): git$X $(QUIET_BUILT_IN)$(RM) $@ && \ @@ -2564,9 +2578,9 @@ config-list.h: generate-configlist.sh config-list.h: Documentation/*config.adoc Documentation/config/*.adoc $(QUIET_GEN)$(SHELL_PATH) ./generate-configlist.sh . $@ -command-list.h: generate-cmdlist.sh command-list.txt +$(COMMAND_LIST_H): generate-cmdlist.sh command-list.txt -command-list.h: $(wildcard Documentation/git*.adoc) +$(COMMAND_LIST_H): $(wildcard Documentation/git*.adoc) $(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \ $(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \ . $@ @@ -2589,10 +2603,10 @@ $(SCRIPT_SH_GEN) $(SCRIPT_LIB) : % : %.sh generate-script.sh GIT-BUILD-OPTIONS G $(QUIET_GEN)./generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \ mv $@+ $@ -git.rc: git.rc.in GIT-VERSION-GEN GIT-VERSION-FILE +git.rc: git.rc.in GIT-VERSION-GEN $(GIT_VERSION_FILE) $(QUIET_GEN)$(call version_gen,"$(shell pwd)",$<,$@) -git.res: git.rc GIT-PREFIX +git.res: git.rc $(GIT_PREFIX) $(QUIET_RC)$(RC) -i $< -o $@ # This makes sure we depend on the NO_PERL setting itself. @@ -2626,8 +2640,8 @@ endif PERL_DEFINES += $(gitexecdir) $(perllibdir) $(localedir) -$(SCRIPT_PERL_GEN): % : %.perl generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER GIT-VERSION-FILE - $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@+" && \ +$(SCRIPT_PERL_GEN): % : %.perl generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER $(GIT_VERSION_FILE) + $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS $(GIT_VERSION_PATH) GIT-PERL-HEADER "$<" "$@+" && \ mv $@+ $@ PERL_DEFINES := $(subst $(space),:,$(PERL_DEFINES)) @@ -2691,7 +2705,7 @@ CONFIGURE_RECIPE = sed -e 's/@GIT_VERSION@/$(GIT_VERSION)/g' \ autoconf -o configure configure.ac+ && \ $(RM) configure.ac+ -configure: configure.ac GIT-VERSION-FILE +configure: configure.ac $(GIT_VERSION_FILE) $(QUIET_GEN)$(CONFIGURE_RECIPE) ifdef AUTOCONFIGURED @@ -2774,6 +2788,14 @@ endif .PHONY: objects objects: $(OBJECTS) +ifdef CARGO_OUT_DIR +OBJECTS := $(addprefix $(CARGO_OUT_DIR)/,$(OBJECTS)) +LIB_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(LIB_OBJS)) +REFTABLE_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(REFTABLE_OBJS)) +XDIFF_OBJS := $(addprefix $(CARGO_OUT_DIR)/,$(XDIFF_OBJS)) +BASIC_CFLAGS += -I$(CARGO_OUT_DIR) +endif + dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d) dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS)))) @@ -2805,10 +2827,15 @@ missing_compdb_dir = compdb_args = endif -$(OBJECTS): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir) +ifdef CARGO_OUT_DIR +$(OBJECTS): $(CARGO_OUT_DIR)/%.o: %.c $(GIT_CFLAGS) $(missing_dep_dirs) $(missing_compdb_dir) + $(QUIET_CC)$(CC) -o $(CARGO_OUT_DIR)/$*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< +else +$(OBJECTS): %.o: %.c $(GIT_CFLAGS) $(missing_dep_dirs) $(missing_compdb_dir) $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< +endif -%.s: %.c GIT-CFLAGS FORCE +%.s: %.c $(GIT_CFLAGS) FORCE $(QUIET_CC)$(CC) -o $@ -S $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $< ifdef USE_COMPUTED_HEADER_DEPENDENCIES @@ -2829,27 +2856,27 @@ compile_commands.json: @if test -s $@+; then mv $@+ $@; else $(RM) $@+; fi endif -exec-cmd.sp exec-cmd.s exec-cmd.o: GIT-PREFIX -exec-cmd.sp exec-cmd.s exec-cmd.o: EXTRA_CPPFLAGS = \ +exec-cmd.sp exec-cmd.s $(call maybe_join_path,$(CARGO_OUT_DIR),exec-cmd.o): $(GIT_PREFIX) +exec-cmd.sp exec-cmd.s $(call maybe_join_path,$(CARGO_OUT_DIR),exec-cmd.o): EXTRA_CPPFLAGS = \ '-DGIT_EXEC_PATH="$(gitexecdir_SQ)"' \ '-DGIT_LOCALE_PATH="$(localedir_relative_SQ)"' \ '-DBINDIR="$(bindir_relative_SQ)"' \ '-DFALLBACK_RUNTIME_PREFIX="$(prefix_SQ)"' -setup.sp setup.s setup.o: GIT-PREFIX -setup.sp setup.s setup.o: EXTRA_CPPFLAGS = \ +setup.sp setup.s $(call maybe_join_path,$(CARGO_OUT_DIR),setup.o): $(GIT_PREFIX) +setup.sp setup.s $(call maybe_join_path,$(CARGO_OUT_DIR),setup.o): EXTRA_CPPFLAGS = \ -DDEFAULT_GIT_TEMPLATE_DIR='"$(template_dir_SQ)"' -config.sp config.s config.o: GIT-PREFIX -config.sp config.s config.o: EXTRA_CPPFLAGS = \ +config.sp config.s $(call maybe_join_path,$(CARGO_OUT_DIR),config.o): $(GIT_PREFIX) +config.sp config.s $(call maybe_join_path,$(CARGO_OUT_DIR),config.o): EXTRA_CPPFLAGS = \ -DETC_GITCONFIG='"$(ETC_GITCONFIG_SQ)"' -attr.sp attr.s attr.o: GIT-PREFIX -attr.sp attr.s attr.o: EXTRA_CPPFLAGS = \ +attr.sp attr.s $(call maybe_join_path,$(CARGO_OUT_DIR),attr.o): $(GIT_PREFIX) +attr.sp attr.s $(call maybe_join_path,$(CARGO_OUT_DIR),attr.o): EXTRA_CPPFLAGS = \ -DETC_GITATTRIBUTES='"$(ETC_GITATTRIBUTES_SQ)"' -gettext.sp gettext.s gettext.o: GIT-PREFIX -gettext.sp gettext.s gettext.o: EXTRA_CPPFLAGS = \ +gettext.sp gettext.s $(call maybe_join_path,$(CARGO_OUT_DIR),gettext.o): $(GIT_PREFIX) +gettext.sp gettext.s $(call maybe_join_path,$(CARGO_OUT_DIR),gettext.o): EXTRA_CPPFLAGS = \ -DGIT_LOCALE_PATH='"$(localedir_relative_SQ)"' http-push.sp http.sp http-walker.sp remote-curl.sp imap-send.sp: SP_EXTRA_FLAGS += \ @@ -2872,7 +2899,7 @@ compat/nedmalloc/nedmalloc.sp compat/nedmalloc/nedmalloc.o: EXTRA_CPPFLAGS = \ compat/nedmalloc/nedmalloc.sp: SP_EXTRA_FLAGS += -Wno-non-pointer-null endif -headless-git.o: compat/win32/headless.c GIT-CFLAGS +headless-git.o: compat/win32/headless.c $(GIT_CFLAGS) $(QUIET_CC)$(CC) $(ALL_CFLAGS) $(COMPAT_CFLAGS) \ -fno-stack-protector -o $@ -c -Wall -Wwrite-strings $< @@ -3116,9 +3143,9 @@ endif NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif -perl/build/lib/%.pm: perl/%.pm generate-perl.sh GIT-BUILD-OPTIONS GIT-VERSION-FILE GIT-PERL-DEFINES +perl/build/lib/%.pm: perl/%.pm generate-perl.sh GIT-BUILD-OPTIONS $(GIT_VERSION_FILE) GIT-PERL-DEFINES $(call mkdir_p_parent_template) - $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@" + $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS $(GIT_VERSION_PATH) GIT-PERL-HEADER "$<" "$@" perl/build/man/man3/Git.3pm: perl/Git.pm $(call mkdir_p_parent_template) @@ -3146,20 +3173,20 @@ cscope: cscope.out TRACK_PREFIX = $(bindir_SQ):$(gitexecdir_SQ):$(template_dir_SQ):$(prefix_SQ):\ $(localedir_SQ) -GIT-PREFIX: FORCE +$(GIT_PREFIX): FORCE @FLAGS='$(TRACK_PREFIX)'; \ - if test x"$$FLAGS" != x"`cat GIT-PREFIX 2>/dev/null`" ; then \ + if test x"$$FLAGS" != x"`cat $(GIT_PREFIX) 2>/dev/null`" ; then \ echo >&2 " * new prefix flags"; \ - echo "$$FLAGS" >GIT-PREFIX; \ + echo "$$FLAGS" >$(GIT_PREFIX); \ fi TRACK_CFLAGS = $(CC):$(subst ','\'',$(ALL_CFLAGS)):$(USE_GETTEXT_SCHEME) -GIT-CFLAGS: FORCE +$(GIT_CFLAGS): FORCE @FLAGS='$(TRACK_CFLAGS)'; \ - if test x"$$FLAGS" != x"`cat GIT-CFLAGS 2>/dev/null`" ; then \ + if test x"$$FLAGS" != x"`cat $(GIT_CFLAGS) 2>/dev/null`" ; then \ echo >&2 " * new build flags"; \ - echo "$$FLAGS" >GIT-CFLAGS; \ + echo "$$FLAGS" >$(GIT_CFLAGS); \ fi TRACK_LDFLAGS = $(subst ','\'',$(ALL_LDFLAGS)) @@ -3751,7 +3778,7 @@ clean: profile-clean coverage-clean cocciclean $(RM) $(FUZZ_PROGRAMS) $(RM) $(SP_OBJ) $(RM) $(HCC) - $(RM) version-def.h + $(RM) $(VERSION_DEF_H) $(RM) -r $(dep_dirs) $(compdb_dir) compile_commands.json $(RM) $(test_bindir_programs) $(RM) -r po/build/ @@ -3774,8 +3801,8 @@ ifndef NO_TCLTK $(MAKE) -C gitk-git clean $(MAKE) -C git-gui clean endif - $(RM) GIT-VERSION-FILE GIT-CFLAGS GIT-LDFLAGS GIT-BUILD-OPTIONS - $(RM) GIT-USER-AGENT GIT-PREFIX + $(RM) $(GIT_VERSION_FILE) $(GIT_CFLAGS) GIT-LDFLAGS GIT-BUILD-OPTIONS + $(RM) $(GIT_USER_AGENT_FILE) $(GIT_PREFIX) $(RM) GIT-SCRIPT-DEFINES GIT-PERL-DEFINES GIT-PERL-HEADER GIT-PYTHON-VARS ifdef MSVC $(RM) $(patsubst %.o,%.o.pdb,$(OBJECTS)) @@ -3937,14 +3964,14 @@ ifdef INCLUDE_LIBGIT_RS all:: libgit-sys libgit-rs endif -LIBGIT_PUB_OBJS += contrib/libgitpub/public_symbol_export.o -LIBGIT_PUB_OBJS += libgit.a -LIBGIT_PUB_OBJS += reftable/libreftable.a -LIBGIT_PUB_OBJS += xdiff/lib.a - -LIBGIT_PARTIAL_EXPORT = contrib/libgitpub/partial_symbol_export.o +LIBGIT_PUB_OBJS += $(CARGO_OUT_DIR)/contrib/libgitpub/public_symbol_export.o +LIBGIT_PUB_OBJS += $(LIB_FILE) +LIBGIT_PUB_OBJS += $(REFTABLE_LIB) +LIBGIT_PUB_OBJS += $(XDIFF_LIB) -LIBGIT_HIDDEN_EXPORT = contrib/libgitpub/hidden_symbol_export.o +LIBGIT_PARTIAL_EXPORT = $(CARGO_OUT_DIR)/contrib/libgitpub/partial_symbol_export.o +LIBGIT_HIDDEN_EXPORT = $(CARGO_OUT_DIR)/contrib/libgitpub/hidden_symbol_export.o +GITPUB_LIB = $(call maybe_join_path,$(CARGO_OUT_DIR),contrib/libgitpub/libgitpub.a) $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LD) -r $^ -o $@ @@ -3952,5 +3979,5 @@ $(LIBGIT_PARTIAL_EXPORT): $(LIBGIT_PUB_OBJS) $(LIBGIT_HIDDEN_EXPORT): $(LIBGIT_PARTIAL_EXPORT) $(OBJCOPY) --localize-hidden $^ $@ -contrib/libgitpub/libgitpub.a: $(LIBGIT_HIDDEN_EXPORT) +$(GITPUB_LIB): $(LIBGIT_HIDDEN_EXPORT) $(AR) $(ARFLAGS) $@ $^ diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index e0d979c196..19407663f5 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -6,7 +6,7 @@ pub fn main() -> std::io::Result<()> { ac.emit_has_path("std::ffi::c_char"); let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); - let git_root = crate_root.join("../.."); + let git_root = crate_root.join("git-src"); let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap()); let make_output = make_cmd::gnu_make() @@ -14,8 +14,9 @@ pub fn main() -> std::io::Result<()> { .env_remove("PROFILE") .current_dir(git_root.clone()) .args([ + &format!("CARGO_OUT_DIR={}", dst.display()), "INCLUDE_LIBGIT_RS=YesPlease", - "contrib/libgitpub/libgitpub.a", + &format!("{}/contrib/libgitpub/libgitpub.a", dst.display()), ]) .output() .expect("Make failed to run"); @@ -26,8 +27,8 @@ pub fn main() -> std::io::Result<()> { String::from_utf8(make_output.stderr).unwrap() ); } - std::fs::copy(git_root.join("contrib/libgitpub/libgitpub.a"), dst.join("libgitpub.a"))?; println!("cargo:rustc-link-search=native={}", dst.display()); + println!("cargo:rustc-link-search=native={}", dst.join("contrib/libgitpub").display()); println!("cargo:rustc-link-lib=gitpub"); println!("cargo:rerun-if-changed={}", git_root.display()); diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src new file mode 120000 index 0000000000..c25bddb6dd --- /dev/null +++ b/contrib/libgit-sys/git-src @@ -0,0 +1 @@ +../.. \ No newline at end of file diff --git a/shared.mak b/shared.mak index 1a99848a95..0dc611dd90 100644 --- a/shared.mak +++ b/shared.mak @@ -127,3 +127,8 @@ GIT_USER_AGENT="$(GIT_USER_AGENT)" \ GIT_VERSION="$(GIT_VERSION_OVERRIDE)" \ $(SHELL_PATH) "$(1)/GIT-VERSION-GEN" "$(1)" "$(2)" "$(3)" endef + +# Apply a path prefix if the prefix is non-empty +define maybe_join_path +$(if $(1),$(1)/$(2),$(2)) +endef -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon @ 2025-03-23 1:46 ` Eric Sunshine 2025-03-24 15:42 ` Junio C Hamano 0 siblings, 1 reply; 21+ messages in thread From: Eric Sunshine @ 2025-03-23 1:46 UTC (permalink / raw) To: Josh Steadmon; +Cc: git, brian m. carlson, Phillip Wood On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote: > Unlike `cargo build`, `cargo package` does not get access to the entire Git repo > containing a Rust crate. Instead, it prepares a directory starting from the > crate root (potentially excluding files, such as those not under version > control, or explicity excluded in the Cargo.toml file). s/explicity/explicitly/ > diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src > @@ -0,0 +1 @@ > +../.. > \ No newline at end of file Meh. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-23 1:46 ` Eric Sunshine @ 2025-03-24 15:42 ` Junio C Hamano 2025-03-25 17:57 ` Josh Steadmon 0 siblings, 1 reply; 21+ messages in thread From: Junio C Hamano @ 2025-03-24 15:42 UTC (permalink / raw) To: Eric Sunshine; +Cc: Josh Steadmon, git, brian m. carlson, Phillip Wood Eric Sunshine <sunshine@sunshineco.com> writes: > On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote: >> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo >> containing a Rust crate. Instead, it prepares a directory starting from the >> crate root (potentially excluding files, such as those not under version >> control, or explicity excluded in the Cargo.toml file). > > s/explicity/explicitly/ > >> diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src >> @@ -0,0 +1 @@ >> +../.. >> \ No newline at end of file > > Meh. https://github.com/git/git/actions/runs/14030831429/job/39278185588#step:3:1 All of the Windows test jobs (not build ones) are broken due to the presence of ../.. symbolic link. Is that ugly hack the only way we can make this work? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-24 15:42 ` Junio C Hamano @ 2025-03-25 17:57 ` Josh Steadmon 2025-03-25 23:08 ` Junio C Hamano 0 siblings, 1 reply; 21+ messages in thread From: Josh Steadmon @ 2025-03-25 17:57 UTC (permalink / raw) To: Junio C Hamano; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood On 2025.03.24 08:42, Junio C Hamano wrote: > Eric Sunshine <sunshine@sunshineco.com> writes: > > > On Fri, Mar 21, 2025 at 6:14 PM Josh Steadmon <steadmon@google.com> wrote: > >> Unlike `cargo build`, `cargo package` does not get access to the entire Git repo > >> containing a Rust crate. Instead, it prepares a directory starting from the > >> crate root (potentially excluding files, such as those not under version > >> control, or explicity excluded in the Cargo.toml file). > > > > s/explicity/explicitly/ > > > >> diff --git a/contrib/libgit-sys/git-src b/contrib/libgit-sys/git-src > >> @@ -0,0 +1 @@ > >> +../.. > >> \ No newline at end of file > > > > Meh. > > https://github.com/git/git/actions/runs/14030831429/job/39278185588#step:3:1 > > All of the Windows test jobs (not build ones) are broken due to the > presence of ../.. symbolic link. > > Is that ugly hack the only way we can make this work? It's the only way I know of to accomplish both: 1) creating a packaged crate with `cargo package` and 2) keeping the top-level clean of any Rust code or configuration. If we're willing to have a Cargo.toml file in the repo root, we could create a "Cargo workspace", but I'm not sure yet if that avoids the same problem with accessing sources outside of the crates themselves. I'll be able to test it out later this week. If the workspace approach doesn't work, the alternatives are: 1) avoid the issue for now; anyone who wants to experiment with libgit-rs can do so by building from source (but it will prevent them from creating their own packaged crates IIUC). 2) move libgit-sys and libgit-rs to separate repos and depend on the Git source via submodules. This is what I've seen done in other -sys crates such as zlib-sys (https://github.com/rust-lang/libz-sys). Of those alternatives, I prefer #1 for now. If we build enough momentum on libification and expanding the coverage of these crates, then we could think about switching to #2. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-25 17:57 ` Josh Steadmon @ 2025-03-25 23:08 ` Junio C Hamano 2025-03-27 18:58 ` Josh Steadmon 0 siblings, 1 reply; 21+ messages in thread From: Junio C Hamano @ 2025-03-25 23:08 UTC (permalink / raw) To: Josh Steadmon; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood Josh Steadmon <steadmon@google.com> writes: > 2) keeping the top-level clean of any Rust code or configuration. > > If we're willing to have a Cargo.toml file in the repo root, ... If it is more like adding a new build configuration file whereever we have Makefile (or meson.build), and is not like we are adding one new file per one existing source file, then I see no reason why we want to avoid adding a few files to the root-level. > ... we could > create a "Cargo workspace", but I'm not sure yet if that avoids the same > problem with accessing sources outside of the crates themselves. I'll be > able to test it out later this week. Yeah, that would probably be a reasonable thing to try. Thanks. > If the workspace approach doesn't work, the alternatives are: > > 1) avoid the issue for now; anyone who wants to experiment with > libgit-rs can do so by building from source (but it will prevent them > from creating their own packaged crates IIUC). > > 2) move libgit-sys and libgit-rs to separate repos and depend on the Git > source via submodules. This is what I've seen done in other -sys crates > such as zlib-sys (https://github.com/rust-lang/libz-sys). > > Of those alternatives, I prefer #1 for now. If we build enough momentum > on libification and expanding the coverage of these crates, then we > could think about switching to #2. Yeah, or putting it another way, #1 would help us gather enough Rust minded folks who are familiar enough to come up with ideas and offer better ways to manage this part of the system. Thanks. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-25 23:08 ` Junio C Hamano @ 2025-03-27 18:58 ` Josh Steadmon 2025-03-29 10:46 ` Junio C Hamano 2025-03-31 14:52 ` Johannes Schindelin 0 siblings, 2 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-27 18:58 UTC (permalink / raw) To: Junio C Hamano; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood On 2025.03.25 16:08, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > 2) keeping the top-level clean of any Rust code or configuration. > > > > If we're willing to have a Cargo.toml file in the repo root, ... > > If it is more like adding a new build configuration file whereever > we have Makefile (or meson.build), and is not like we are adding one > new file per one existing source file, then I see no reason why we > want to avoid adding a few files to the root-level. > > > ... we could > > create a "Cargo workspace", but I'm not sure yet if that avoids the same > > problem with accessing sources outside of the crates themselves. I'll be > > able to test it out later this week. > > Yeah, that would probably be a reasonable thing to try. Thanks. > > > > If the workspace approach doesn't work, the alternatives are: > > > > 1) avoid the issue for now; anyone who wants to experiment with > > libgit-rs can do so by building from source (but it will prevent them > > from creating their own packaged crates IIUC). > > > > 2) move libgit-sys and libgit-rs to separate repos and depend on the Git > > source via submodules. This is what I've seen done in other -sys crates > > such as zlib-sys (https://github.com/rust-lang/libz-sys). > > > > Of those alternatives, I prefer #1 for now. If we build enough momentum > > on libification and expanding the coverage of these crates, then we > > could think about switching to #2. > > Yeah, or putting it another way, #1 would help us gather enough Rust > minded folks who are familiar enough to come up with ideas and offer > better ways to manage this part of the system. > > Thanks. Unfortunately creating a workspace doesn't provide access to the top-level source. Symlinks seem to be the only recommended approach [1] for embedded crates, but since that breaks Windows CI let's just drop the series for now. [1] https://users.rust-lang.org/t/including-files-from-parent-directory-in-package/88969 ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-27 18:58 ` Josh Steadmon @ 2025-03-29 10:46 ` Junio C Hamano 2025-03-31 14:52 ` Johannes Schindelin 1 sibling, 0 replies; 21+ messages in thread From: Junio C Hamano @ 2025-03-29 10:46 UTC (permalink / raw) To: Josh Steadmon; +Cc: Eric Sunshine, git, brian m. carlson, Phillip Wood Josh Steadmon <steadmon@google.com> writes: > Unfortunately creating a workspace doesn't provide access to the > top-level source. Symlinks seem to be the only recommended approach [1] > for embedded crates, but since that breaks Windows CI let's just drop > the series for now. Understood. Thanks. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree 2025-03-27 18:58 ` Josh Steadmon 2025-03-29 10:46 ` Junio C Hamano @ 2025-03-31 14:52 ` Johannes Schindelin 1 sibling, 0 replies; 21+ messages in thread From: Johannes Schindelin @ 2025-03-31 14:52 UTC (permalink / raw) To: Josh Steadmon Cc: Junio C Hamano, Eric Sunshine, git, brian m. carlson, Phillip Wood Hi Josh, On Thu, 27 Mar 2025, Josh Steadmon wrote: > On 2025.03.25 16:08, Junio C Hamano wrote: > > Josh Steadmon <steadmon@google.com> writes: > > > > > 2) keeping the top-level clean of any Rust code or configuration. > > > > > > If we're willing to have a Cargo.toml file in the repo root, ... > > > > If it is more like adding a new build configuration file whereever > > we have Makefile (or meson.build), and is not like we are adding one > > new file per one existing source file, then I see no reason why we > > want to avoid adding a few files to the root-level. > > > > > ... we could > > > create a "Cargo workspace", but I'm not sure yet if that avoids the same > > > problem with accessing sources outside of the crates themselves. I'll be > > > able to test it out later this week. > > > > Yeah, that would probably be a reasonable thing to try. Thanks. > > > > > > > If the workspace approach doesn't work, the alternatives are: > > > > > > 1) avoid the issue for now; anyone who wants to experiment with > > > libgit-rs can do so by building from source (but it will prevent them > > > from creating their own packaged crates IIUC). > > > > > > 2) move libgit-sys and libgit-rs to separate repos and depend on the Git > > > source via submodules. This is what I've seen done in other -sys crates > > > such as zlib-sys (https://github.com/rust-lang/libz-sys). > > > > > > Of those alternatives, I prefer #1 for now. If we build enough momentum > > > on libification and expanding the coverage of these crates, then we > > > could think about switching to #2. > > > > Yeah, or putting it another way, #1 would help us gather enough Rust > > minded folks who are familiar enough to come up with ideas and offer > > better ways to manage this part of the system. > > > > Thanks. > > Unfortunately creating a workspace doesn't provide access to the > top-level source. Symlinks seem to be the only recommended approach [1] > for embedded crates, but since that breaks Windows CI let's just drop > the series for now. > > [1] https://users.rust-lang.org/t/including-files-from-parent-directory-in-package/88969 If you need symbolic linkson Windows in CI, please set `MSYS=winsymlinks:nativestrict` like it is done here: https://github.com/git-for-windows/git/blob/4ca71ba5311a8f1bafbf002e97e076f15dcfc15b/t/t2040-checkout-symlink-attr.sh#L8-L10. It might fail, though, if it requires _Git_ to be able to create symbolic links because I have not yet managed to upstream the patches to implement that. We could fast-track support for `readlink()` without the other parts (https://github.com/git-for-windows/git/commit/86420a1b84d1), but last time I checked, Git's test suite did not pass under `MSYS=winsymlinks:nativestrict` for some reasons I was unable to analyze for lack of time. Ciao, Johannes ^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood Cargo provides GNU Make flags that can be used to connect to its jobserver for managing parallel builds. They are not automatically passed to any `make` invocations, so let's add that to our build.rs script. Signed-off-by: Josh Steadmon <steadmon@google.com> --- contrib/libgit-sys/build.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 19407663f5..1237c9a37f 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -8,9 +8,11 @@ pub fn main() -> std::io::Result<()> { let crate_root = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); let git_root = crate_root.join("git-src"); let dst = PathBuf::from(env::var_os("OUT_DIR").unwrap()); + let makeflags = env::var_os("CARGO_MAKEFLAGS").unwrap(); let make_output = make_cmd::gnu_make() .env("DEVELOPER", "1") + .env("MAKEFLAGS", &makeflags) .env_remove("PROFILE") .current_dir(git_root.clone()) .args([ -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon ` (2 preceding siblings ...) 2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood We can avoid copying tens of megabytes of unnecessary source files by excluding a few directories which are not needed to compile libgitpub.a. This helps us stay below crates.io's 10 MB size limit. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 2 +- contrib/libgit-sys/Cargo.toml | 12 ++++++++++++ contrib/libgit-sys/build.rs | 3 +++ 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 91677448ba..1574fc76e2 100644 --- a/Makefile +++ b/Makefile @@ -3500,13 +3500,13 @@ coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE) # "Sub"-Makefiles, not really because they can't be run stand-alone, # only there to contain directory-specific rules and variables +ifndef NO_GITWEB ## gitweb/Makefile inclusion: MAK_DIR_GITWEB = gitweb/ include gitweb/Makefile .PHONY: gitweb gitweb: $(MAK_DIR_GITWEB_ALL) -ifndef NO_GITWEB all:: gitweb endif diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml index e0623022c3..1c1efb6211 100644 --- a/contrib/libgit-sys/Cargo.toml +++ b/contrib/libgit-sys/Cargo.toml @@ -7,6 +7,18 @@ links = "gitpub" rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from # version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/ description = "Native bindings to a portion of libgit" +exclude = [ + "git-src/.github", + "git-src/Documentation", + "git-src/ci", + "git-src/git-gui", + "git-src/gitk-git", + "git-src/gitweb", + "git-src/oss-fuzz", + "git-src/perl", + "git-src/po", + "git-src/t" +] [lib] path = "src/lib.rs" diff --git a/contrib/libgit-sys/build.rs b/contrib/libgit-sys/build.rs index 1237c9a37f..7e1244fcd5 100644 --- a/contrib/libgit-sys/build.rs +++ b/contrib/libgit-sys/build.rs @@ -17,6 +17,9 @@ pub fn main() -> std::io::Result<()> { .current_dir(git_root.clone()) .args([ &format!("CARGO_OUT_DIR={}", dst.display()), + "NO_GITWEB=YesPlease", + "NO_PERL=YesPlease", + "NO_TCLTK=YesPlease", "INCLUDE_LIBGIT_RS=YesPlease", &format!("{}/contrib/libgitpub/libgitpub.a", dst.display()), ]) -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields 2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon ` (3 preceding siblings ...) 2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon @ 2025-03-21 22:14 ` Josh Steadmon 4 siblings, 0 replies; 21+ messages in thread From: Josh Steadmon @ 2025-03-21 22:14 UTC (permalink / raw) To: git; +Cc: brian m. carlson, Phillip Wood Add license and description fields for libgit-sys and libgit-rs. These are required to publish our crates on crates.io. Signed-off-by: Josh Steadmon <steadmon@google.com> --- contrib/libgit-rs/Cargo.toml | 2 ++ contrib/libgit-sys/Cargo.toml | 1 + 2 files changed, 3 insertions(+) diff --git a/contrib/libgit-rs/Cargo.toml b/contrib/libgit-rs/Cargo.toml index c3289e69db..9747ba3cb4 100644 --- a/contrib/libgit-rs/Cargo.toml +++ b/contrib/libgit-rs/Cargo.toml @@ -5,6 +5,8 @@ edition = "2021" build = "build.rs" rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from # version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/ +description = "Proof-of-concept Rust API for a portion of libgit" +license = "GPL-2.0-only" [lib] diff --git a/contrib/libgit-sys/Cargo.toml b/contrib/libgit-sys/Cargo.toml index 1c1efb6211..cfbcca9c8c 100644 --- a/contrib/libgit-sys/Cargo.toml +++ b/contrib/libgit-sys/Cargo.toml @@ -7,6 +7,7 @@ links = "gitpub" rust-version = "1.63" # TODO: Once we hit 1.84 or newer, we may want to remove Cargo.lock from # version control. See https://lore.kernel.org/git/Z47jgK-oMjFRSslr@tapette.crustytoothpaste.net/ description = "Native bindings to a portion of libgit" +license = "GPL-2.0-only" exclude = [ "git-src/.github", "git-src/Documentation", -- 2.49.0.395.g12beb8f557-goog ^ permalink raw reply related [flat|nested] 21+ messages in thread
end of thread, other threads:[~2025-03-31 14:52 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-03-18 23:24 [RFC PATCH v1 0/4] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 1/4] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 2/4] libgit-sys: add symlink to git repo root and clean after build Josh Steadmon
2025-03-19 22:23 ` Josh Steadmon
2025-03-20 11:10 ` Phillip Wood
2025-03-21 19:49 ` Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 3/4] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
2025-03-18 23:24 ` [RFC PATCH v1 4/4] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 0/5] Fix `cargo package` for libgit-sys Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 1/5] libgitpub: move to separate contrib/ directory Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 2/5] libgit-sys: add symlink to git repo root and build out of tree Josh Steadmon
2025-03-23 1:46 ` Eric Sunshine
2025-03-24 15:42 ` Junio C Hamano
2025-03-25 17:57 ` Josh Steadmon
2025-03-25 23:08 ` Junio C Hamano
2025-03-27 18:58 ` Josh Steadmon
2025-03-29 10:46 ` Junio C Hamano
2025-03-31 14:52 ` Johannes Schindelin
2025-03-21 22:14 ` [PATCH v2 3/5] libgit-sys: parallelize build with Cargo's jobserver Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 4/5] libgit-sys: exclude unnecessary directories in git-src Josh Steadmon
2025-03-21 22:14 ` [PATCH v2 5/5] libgit-{sys,rs}: add license and description fields Josh Steadmon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).