[Buildroot] [PATCH 0/2] Fix .gitignore in CANONICAL_O breaking Python packages

[Buildroot] [PATCH 0/2] Fix .gitignore in CANONICAL_O breaking Python packages

[Brandon Maier via buildroot]

I attempted to make the existing gitignore logic work with the
python-jsonschema-specifications, but I haven't found a method that
works.

To unbreak the package, revert the gitignore change. I included a second
patch that just ignores all directories that match 'output*', as this at
least allows for multiple output directories but didn't break
python-jsonschema-specifications.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
---
Brandon Maier (2):
      Revert "Create a .gitignore file in the CANONICAL_O directory"
      gitignore: ignore multiple output directories

 .gitignore | 1 +
 Makefile   | 3 ---
 2 files changed, 1 insertion(+), 3 deletions(-)
---
base-commit: 8ce17e304b02c6969e4fd51f0bde97798d49eb3b
change-id: 20240724-revert-output-gitignore-38e543e10cd6

Best regards,
-- 
Brandon Maier <brandon.maier@collins.com>

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[Brandon Maier via buildroot]

The packages 'python-jsonschema-specifications', 'pydantic-core', and
likely other packages based on Python maturin silently stopped
installing files to the host and target directories after commit
a14c862c08.

A similar issue was reported by the Conda project[1]. It seems some
build tools scan for gitignore files to decide what files to build
and install.

I attempted various combinations of gitignore patterns including
mimicking the Conda project's gitignore[2], but the only thing that has
worked reliably is the original `/output` ignore in the root Buildroot
directory.

This reverts commit a14c862c08865e053d5fce90a0d0323b8f9e4bc0.

[1] https://github.com/PyO3/maturin/issues/1911
[2] https://github.com/conda-forge/conda-smithy/blob/main/conda_smithy/feedstock_content/.gitignore
Reported-by: Bryan Brattlof <bb@ti.com>
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
---
 .gitignore | 1 +
 Makefile   | 3 ---
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/.gitignore b/.gitignore
index 14b7ca5040..bb02d9f572 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
+/output
 /dl
 /.auto.deps
 /.config.cmd
diff --git a/Makefile b/Makefile
index 0432a014ab..f51e8f690f 100644
--- a/Makefile
+++ b/Makefile
@@ -63,9 +63,6 @@ ifneq ($(findstring @,$(CANONICAL_O)),)
 $(error The build directory can not contain a '@')
 endif
 
-# Create a default .gitignore file that ignores everything
-$(shell echo "*" > "$(CANONICAL_O)/.gitignore")
-
 CANONICAL_CURDIR = $(realpath $(CURDIR))
 
 REQ_UMASK = 0022

-- 
2.45.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] gitignore: ignore multiple output directories

[Brandon Maier via buildroot]

Developers may have multiple output directories when building multiple
boards in parallel. A previous commit attempted to gitignore new output
directories automatically but broke some things (commit a14c862c08
"Create a .gitignore file in the CANONICAL_O directory").

To provide some support, ignore anything in the Buildroot directory that
starts with "output". This is a bit more cumbersome, but doesn't break
packages based on Python maturin.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
---
 .gitignore | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index bb02d9f572..ec5768d053 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,4 @@
-/output
+/output*
 /dl
 /.auto.deps
 /.config.cmd

-- 
2.45.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[Thomas Petazzoni via buildroot]

Hello Brandon,

On Wed, 24 Jul 2024 21:23:20 +0000
Brandon Maier <brandon.maier@collins.com> wrote:

> The packages 'python-jsonschema-specifications', 'pydantic-core', and
> likely other packages based on Python maturin silently stopped
> installing files to the host and target directories after commit
> a14c862c08.
> 
> A similar issue was reported by the Conda project[1]. It seems some
> build tools scan for gitignore files to decide what files to build
> and install.
> 
> I attempted various combinations of gitignore patterns including
> mimicking the Conda project's gitignore[2], but the only thing that has
> worked reliably is the original `/output` ignore in the root Buildroot
> directory.

Thanks for working on this. However, it's really messed up. Why would
these tools be bothered by a .gitignore file? Do you have some more
details?

I think I'm going to apply the revert to fix the immediate build
issues, but I would really really like to fix the real problem, and
allow us to generate this .gitignore file in the output directory.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[James Hilliard]

On Wed, Jul 24, 2024 at 11:51 PM Thomas Petazzoni via buildroot
<buildroot@buildroot.org> wrote:
>
> Hello Brandon,
>
> On Wed, 24 Jul 2024 21:23:20 +0000
> Brandon Maier <brandon.maier@collins.com> wrote:
>
> > The packages 'python-jsonschema-specifications', 'pydantic-core', and
> > likely other packages based on Python maturin silently stopped
> > installing files to the host and target directories after commit
> > a14c862c08.
> >
> > A similar issue was reported by the Conda project[1]. It seems some
> > build tools scan for gitignore files to decide what files to build
> > and install.
> >
> > I attempted various combinations of gitignore patterns including
> > mimicking the Conda project's gitignore[2], but the only thing that has
> > worked reliably is the original `/output` ignore in the root Buildroot
> > directory.
>
> Thanks for working on this. However, it's really messed up. Why would
> these tools be bothered by a .gitignore file? Do you have some more
> details?

So it's the exact same reason ripgrep is affected by .gitignore files,
maturin actually uses the ignore crate from the ripgrep project:
https://crates.io/crates/ignore

Note that the ignore crate doesn't entirely correctly replicate git's
ignore logic as otherwise this shouldn't have triggered any issues.

>
> I think I'm going to apply the revert to fix the immediate build
> issues, but I would really really like to fix the real problem, and
> allow us to generate this .gitignore file in the output directory.

Yeah, I've been working on a fix but it has some issues, maybe
someone can help fix the tests:
https://github.com/PyO3/maturin/pull/2158

>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[Brandon Maier via buildroot]

Hi Thomas, James,

On Thu Jul 25, 2024 at 6:04 AM UTC, James Hilliard wrote:
> On Wed, Jul 24, 2024 at 11:51 PM Thomas Petazzoni via buildroot
> <buildroot@buildroot.org> wrote:
> >
> > Hello Brandon,
> >
> > On Wed, 24 Jul 2024 21:23:20 +0000
> > Brandon Maier <brandon.maier@collins.com> wrote:
> >
> > > The packages 'python-jsonschema-specifications', 'pydantic-core', and
> > > likely other packages based on Python maturin silently stopped
> > > installing files to the host and target directories after commit
> > > a14c862c08.
> > >
> > > A similar issue was reported by the Conda project[1]. It seems some
> > > build tools scan for gitignore files to decide what files to build
> > > and install.
> > >
> > > I attempted various combinations of gitignore patterns including
> > > mimicking the Conda project's gitignore[2], but the only thing that has
> > > worked reliably is the original `/output` ignore in the root Buildroot
> > > directory.
> >
> > Thanks for working on this. However, it's really messed up. Why would
> > these tools be bothered by a .gitignore file? Do you have some more
> > details?
>
> So it's the exact same reason ripgrep is affected by .gitignore files,
> maturin actually uses the ignore crate from the ripgrep project:
> https://crates.io/crates/ignore
>
> Note that the ignore crate doesn't entirely correctly replicate git's
> ignore logic as otherwise this shouldn't have triggered any issues.

It does seem to replicate git's ignore logic correctly. The issue is
that we download python-jsonschema as a tarball which does not include
the '.git' directory. So when `maturin` tries to use gitignore's, it
does do what git would do, it searches upwards until it finds
Buildroot's '.git' directory and treats it as the repo.

It seems to me then the fix is one of
- Buildroot needs to switch to using the git repo directly
- Buildroot needs to create an empty git repo in build/python-jsonschema/
- Maturin needs to stop searching for git repos at its pyproject
  directory, which James' Pull Request is doing

And I agree fixing it in Maturin is the correct place. As that will also
fix it for anyone using the tarballs, like Conda. Though it does make me
paranoid about what other build tools out there might be expecting a git
repo and then silently falling back to the Buildroot repo...

What was surprising to me is that the `/output` pattern in Buildroot's
root gitignore doesn't cause this problem. This appears to be a bug in
ripgrep's logic, and I found this long thread that seems related. So if
this ever gets fixed then these packages will be broken again.

https://github.com/BurntSushi/ripgrep/issues/829

>
> >
> > I think I'm going to apply the revert to fix the immediate build
> > issues, but I would really really like to fix the real problem, and
> > allow us to generate this .gitignore file in the output directory.
>
> Yeah, I've been working on a fix but it has some issues, maybe
> someone can help fix the tests:
> https://github.com/PyO3/maturin/pull/2158

Thank you James for looking at this so quickly,
Brandon

>
> >
> > Thomas
> > --
> > Thomas Petazzoni, co-owner and CEO, Bootlin
> > Embedded Linux and Kernel engineering and training
> > https://bootlin.com
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[James Hilliard]

On Thu, Jul 25, 2024 at 9:01 AM Brandon Maier <brandon.maier@collins.com> wrote:
>
> Hi Thomas, James,
>
> On Thu Jul 25, 2024 at 6:04 AM UTC, James Hilliard wrote:
> > On Wed, Jul 24, 2024 at 11:51 PM Thomas Petazzoni via buildroot
> > <buildroot@buildroot.org> wrote:
> > >
> > > Hello Brandon,
> > >
> > > On Wed, 24 Jul 2024 21:23:20 +0000
> > > Brandon Maier <brandon.maier@collins.com> wrote:
> > >
> > > > The packages 'python-jsonschema-specifications', 'pydantic-core', and
> > > > likely other packages based on Python maturin silently stopped
> > > > installing files to the host and target directories after commit
> > > > a14c862c08.
> > > >
> > > > A similar issue was reported by the Conda project[1]. It seems some
> > > > build tools scan for gitignore files to decide what files to build
> > > > and install.
> > > >
> > > > I attempted various combinations of gitignore patterns including
> > > > mimicking the Conda project's gitignore[2], but the only thing that has
> > > > worked reliably is the original `/output` ignore in the root Buildroot
> > > > directory.
> > >
> > > Thanks for working on this. However, it's really messed up. Why would
> > > these tools be bothered by a .gitignore file? Do you have some more
> > > details?
> >
> > So it's the exact same reason ripgrep is affected by .gitignore files,
> > maturin actually uses the ignore crate from the ripgrep project:
> > https://crates.io/crates/ignore
> >
> > Note that the ignore crate doesn't entirely correctly replicate git's
> > ignore logic as otherwise this shouldn't have triggered any issues.
>
> It does seem to replicate git's ignore logic correctly. The issue is
> that we download python-jsonschema as a tarball which does not include
> the '.git' directory. So when `maturin` tries to use gitignore's, it
> does do what git would do, it searches upwards until it finds
> Buildroot's '.git' directory and treats it as the repo.

If it replicated git's ignore logic correctly it would have been broken
before this change already. AFAIU we're essentially relying on a
bug in maturins git ignore logic not parsing the `/output` pattern in
buildroot's gitignore.

>
> It seems to me then the fix is one of
> - Buildroot needs to switch to using the git repo directly

This is the wrong approach IMO.

> - Buildroot needs to create an empty git repo in build/python-jsonschema/

This seems to be pretty hacky, but might be an ok workaround.

> - Maturin needs to stop searching for git repos at its pyproject
>   directory, which James' Pull Request is doing
>
> And I agree fixing it in Maturin is the correct place. As that will also
> fix it for anyone using the tarballs, like Conda. Though it does make me
> paranoid about what other build tools out there might be expecting a git
> repo and then silently falling back to the Buildroot repo...

Poetry had a similar bug I had fixed a little while ago, but it was using
real git and not some custom ignore library like maturin does so the
fix was a bit simpler:
https://github.com/python-poetry/poetry-core/pull/611

>
> What was surprising to me is that the `/output` pattern in Buildroot's
> root gitignore doesn't cause this problem. This appears to be a bug in
> ripgrep's logic, and I found this long thread that seems related. So if
> this ever gets fixed then these packages will be broken again.

This is what I mean by it doesn't replicate git's ignore logic correctly.

>
> https://github.com/BurntSushi/ripgrep/issues/829
>
> >
> > >
> > > I think I'm going to apply the revert to fix the immediate build
> > > issues, but I would really really like to fix the real problem, and
> > > allow us to generate this .gitignore file in the output directory.
> >
> > Yeah, I've been working on a fix but it has some issues, maybe
> > someone can help fix the tests:
> > https://github.com/PyO3/maturin/pull/2158
>
> Thank you James for looking at this so quickly,
> Brandon
>
> >
> > >
> > > Thomas
> > > --
> > > Thomas Petazzoni, co-owner and CEO, Bootlin
> > > Embedded Linux and Kernel engineering and training
> > > https://bootlin.com
> > > _______________________________________________
> > > buildroot mailing list
> > > buildroot@buildroot.org
> > > https://lists.buildroot.org/mailman/listinfo/buildroot
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[Brandon Maier via buildroot]

Hi Thomas,

On Thu Jul 25, 2024 at 5:51 AM UTC, Thomas Petazzoni via buildroot wrote:
> Hello Brandon,
>
> On Wed, 24 Jul 2024 21:23:20 +0000
> Brandon Maier <brandon.maier@collins.com> wrote:
>
> > The packages 'python-jsonschema-specifications', 'pydantic-core', and
> > likely other packages based on Python maturin silently stopped
> > installing files to the host and target directories after commit
> > a14c862c08.
> >
> > A similar issue was reported by the Conda project[1]. It seems some
> > build tools scan for gitignore files to decide what files to build
> > and install.
> >
> > I attempted various combinations of gitignore patterns including
> > mimicking the Conda project's gitignore[2], but the only thing that has
> > worked reliably is the original `/output` ignore in the root Buildroot
> > directory.
>
> Thanks for working on this. However, it's really messed up. Why would
> these tools be bothered by a .gitignore file? Do you have some more
> details?

I did discover I was wrong about python-jsonschema-specifications, it
does not use Maturin, it uses Hatchling. Hatchling also has a gitignore
related bug. I opened a github issue here with more detailed info.

https://github.com/pypa/hatch/issues/1641

>
> I think I'm going to apply the revert to fix the immediate build
> issues, but I would really really like to fix the real problem, and
> allow us to generate this .gitignore file in the output directory.

Same, hopefully we can get these issues worked out and try again :)

Thanks,
Brandon

>
> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[Thomas Petazzoni via buildroot]

On Wed, 24 Jul 2024 21:23:20 +0000
Brandon Maier via buildroot <buildroot@buildroot.org> wrote:

> The packages 'python-jsonschema-specifications', 'pydantic-core', and
> likely other packages based on Python maturin silently stopped
> installing files to the host and target directories after commit
> a14c862c08.
> 
> A similar issue was reported by the Conda project[1]. It seems some
> build tools scan for gitignore files to decide what files to build
> and install.
> 
> I attempted various combinations of gitignore patterns including
> mimicking the Conda project's gitignore[2], but the only thing that has
> worked reliably is the original `/output` ignore in the root Buildroot
> directory.
> 
> This reverts commit a14c862c08865e053d5fce90a0d0323b8f9e4bc0.
> 
> [1] https://github.com/PyO3/maturin/issues/1911
> [2] https://github.com/conda-forge/conda-smithy/blob/main/conda_smithy/feedstock_content/.gitignore
> Reported-by: Bryan Brattlof <bb@ti.com>
> Signed-off-by: Brandon Maier <brandon.maier@collins.com>
> ---
>  .gitignore | 1 +
>  Makefile   | 3 ---
>  2 files changed, 1 insertion(+), 3 deletions(-)

Thanks, both applied. Of course, I'm not a huge fan of PATCH 2/2,
because there's really no reason for output folders to be called
"output<something>", but until we can create the .gitignore file in the
output folder itself, I guess this is a reasonable workaround to allow
people to create multiple output directories that are ignored.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] Revert "Create a .gitignore file in the CANONICAL_O directory"

[James Hilliard]

On Thu, Jul 25, 2024 at 1:34 PM Brandon Maier via buildroot
<buildroot@buildroot.org> wrote:
>
> Hi Thomas,
>
> On Thu Jul 25, 2024 at 5:51 AM UTC, Thomas Petazzoni via buildroot wrote:
> > Hello Brandon,
> >
> > On Wed, 24 Jul 2024 21:23:20 +0000
> > Brandon Maier <brandon.maier@collins.com> wrote:
> >
> > > The packages 'python-jsonschema-specifications', 'pydantic-core', and
> > > likely other packages based on Python maturin silently stopped
> > > installing files to the host and target directories after commit
> > > a14c862c08.
> > >
> > > A similar issue was reported by the Conda project[1]. It seems some
> > > build tools scan for gitignore files to decide what files to build
> > > and install.
> > >
> > > I attempted various combinations of gitignore patterns including
> > > mimicking the Conda project's gitignore[2], but the only thing that has
> > > worked reliably is the original `/output` ignore in the root Buildroot
> > > directory.
> >
> > Thanks for working on this. However, it's really messed up. Why would
> > these tools be bothered by a .gitignore file? Do you have some more
> > details?
>
> I did discover I was wrong about python-jsonschema-specifications, it
> does not use Maturin, it uses Hatchling. Hatchling also has a gitignore
> related bug. I opened a github issue here with more detailed info.
>
> https://github.com/pypa/hatch/issues/1641

Possible fix for the hatch bug:
https://github.com/pypa/hatch/pull/1643

>
> >
> > I think I'm going to apply the revert to fix the immediate build
> > issues, but I would really really like to fix the real problem, and
> > allow us to generate this .gitignore file in the output directory.
>
> Same, hopefully we can get these issues worked out and try again :)
>
> Thanks,
> Brandon
>
> >
> > Thomas
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot