linux-media checkout failing for KernelCI

linux-media checkout failing for KernelCI

[Gustavo Padovan]

Hi Media maintainers,

We are struggling to use the media tree, in the same way we usually 
virtually all other trees in KernelCI. The following commands gives 
KernelCI a fatal error.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git remote add media https://git.linuxtv.org/media_tree.git
git remote update media
git fetch --tags https://git.linuxtv.org/media_tree.git
fatal: Couldn't find remote ref HEAD

We are in the process of enabling a lot of testing for media using 
fluster/gst and the ChromeOS Tast PlatformDecoder tests. I am not a git 
behavior expert, so we want to start this thread to discuss possible 
solutions as the media tree is for some reason deviating from some tacit 
standard.

Thoughts?

Best,

- Gus

-- 
Gustavo Padovan
Kernel Lead

Collabora Ltd.
Platinum Building, St John's Innovation Park
Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: linux-media checkout failing for KernelCI

[Gustavo Padovan]

...linuxtv-ci mailing-list rejected me for not being a member, so moving 
to the media list.

On 5/17/24 09:41, Gustavo Padovan wrote:
>
> Hi Media maintainers,
>
> We are struggling to use the media tree, in the same way we usually 
> virtually all other trees in KernelCI. The following commands gives 
> KernelCI a fatal error.
>
> git clone 
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> cd linux
> git remote add media https://git.linuxtv.org/media_tree.git
> git remote update media
> git fetch --tags https://git.linuxtv.org/media_tree.git
> fatal: Couldn't find remote ref HEAD
>
> We are in the process of enabling a lot of testing for media using 
> fluster/gst and the ChromeOS Tast PlatformDecoder tests. I am not a 
> git behavior expert, so we want to start this thread to discuss 
> possible solutions as the media tree is for some reason deviating from 
> some tacit standard.
>
> Thoughts?
>
> Best,
>
> - Gus
>

-- 
Gustavo Padovan
Kernel Lead

Collabora Ltd.
Platinum Building, St John's Innovation Park
Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: linux-media checkout failing for KernelCI

[Guenter Roeck]

On Fri, May 17, 2024 at 6:41 AM Gustavo Padovan <gus@collabora.com> wrote:
>
>
> Hi Media maintainers,
>
> We are struggling to use the media tree, in the same way we usually
> virtually all other trees in KernelCI. The following commands gives
> KernelCI a fatal error.
>
> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> cd linux
> git remote add media https://git.linuxtv.org/media_tree.git
> git remote update media
> git fetch --tags https://git.linuxtv.org/media_tree.git

Why not "git fetch --tags media" instead of the last two commands ?
That is working for me, and it is what I always use when fetching
remote repositories.

Thanks,
Guenter

> fatal: Couldn't find remote ref HEAD
>
> We are in the process of enabling a lot of testing for media using
> fluster/gst and the ChromeOS Tast PlatformDecoder tests. I am not a git
> behavior expert, so we want to start this thread to discuss possible
> solutions as the media tree is for some reason deviating from some tacit
> standard.
>
> Thoughts?
>
> Best,
>
> - Gus
>
> --
> Gustavo Padovan
> Kernel Lead
>
> Collabora Ltd.
> Platinum Building, St John's Innovation Park
> Cambridge CB4 0DS, UK
> Registered in England & Wales, no. 5513718
>
>

Re: linux-media checkout failing for KernelCI

[Gustavo Padovan]

On 5/17/24 11:29, Guenter Roeck wrote:
> On Fri, May 17, 2024 at 6:41 AM Gustavo Padovan <gus@collabora.com> wrote:
>>
>> Hi Media maintainers,
>>
>> We are struggling to use the media tree, in the same way we usually
>> virtually all other trees in KernelCI. The following commands gives
>> KernelCI a fatal error.
>>
>> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> cd linux
>> git remote add media https://git.linuxtv.org/media_tree.git
>> git remote update media
>> git fetch --tags https://git.linuxtv.org/media_tree.git
> Why not "git fetch --tags media" instead of the last two commands ?
> That is working for me, and it is what I always use when fetching
> remote repositories.

Hmm. Maybe we were just being newbies with the way KernelCI does it. :)

-- 
Gustavo Padovan
Kernel Lead

Collabora Ltd.
Platinum Building, St John's Innovation Park
Cambridge CB4 0DS, UK
Registered in England & Wales, no. 5513718

Re: linux-media checkout failing for KernelCI

[Mauro Carvalho Chehab]

Em Fri, 17 May 2024 09:45:53 -0400
Gustavo Padovan <gus@collabora.com> escreveu:

> ...linuxtv-ci mailing-list rejected me for not being a member, so moving 
> to the media list.
> 
> On 5/17/24 09:41, Gustavo Padovan wrote:
> >
> > Hi Media maintainers,
> >
> > We are struggling to use the media tree, in the same way we usually 
> > virtually all other trees in KernelCI. The following commands gives 
> > KernelCI a fatal error.
> >
> > git clone 
> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > cd linux
> > git remote add media https://git.linuxtv.org/media_tree.git
> > git remote update media
> > git fetch --tags https://git.linuxtv.org/media_tree.git
> > fatal: Couldn't find remote ref HEAD

First of all, the best is to use, instead, media_stage.git, as this
contains the most recent patches. You should notice, however, that
we might need to rebase media_stage.git. That doesn't happen often,
but if you use it for CI, just in case, be sure that it will do a

	git reset --hard FETCH_HEAD

To avoid potential problems if a rebase is needed.

We periodically update media_tree.git as well (which never rebases),
but the update is manual. I usually do it once per week between
-rc1 to -rc7, by the time I handle myself pull requests.

For CI, the best is to use media_stage.git.

-

With regards to merge/fetch, please use the git protocol
whenever possible, e. g.:

	git://linuxtv.org/media_tree.git
	git://linuxtv.org/media_stage.git

you may, alternatively use:

	https://linuxtv.org/git/media_tree.git
	https://linuxtv.org/git/media_stage.git

The 4 above url's are handled by git daemon.

If you use, instead:

	https://git.linuxtv.org/media_tree.git

It will use cgit, which will try to do the right thing, but
results may not be consistent, as the main goal for cgit is to
provide a web visual interface (which is IMO a *lot* better than
what git??b provides).

The problem with https is that it may require a hook during
push time to update https references:

	git update-server-info

Currently, we have such hook enabled for media_tree.git and
media_stage.git. I'm not sure if this is still required for
cgit and git-daemon, though.

If one could test it, that would be nice.

We may migrate some day from git-daemon/cgit to something more modern,
like gitlab, but I'd like to keep a patch-based view like the one
provided by cgit, as IMO showing the Kernel tree by default doesn't
really help for our needs: showing the basis of the Kernel tree as
the initial view means nothing, as one can't tell if new patches were
added or not just looking into it.

> > We are in the process of enabling a lot of testing for media using 
> > fluster/gst and the ChromeOS Tast PlatformDecoder tests. I am not a 
> > git behavior expert, so we want to start this thread to discuss 
> > possible solutions as the media tree is for some reason deviating from 
> > some tacit standard.
> >
> > Thoughts?
> >
> > Best,
> >
> > - Gus
> >  
> 

Re: linux-media checkout failing for KernelCI

[Mauro Carvalho Chehab]

Em Fri, 17 May 2024 19:56:27 +0100
Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:

> Em Fri, 17 May 2024 09:45:53 -0400
> Gustavo Padovan <gus@collabora.com> escreveu:
> 
> > ...linuxtv-ci mailing-list rejected me for not being a member, so moving 
> > to the media list.
> > 
> > On 5/17/24 09:41, Gustavo Padovan wrote:  
> > >
> > > Hi Media maintainers,
> > >
> > > We are struggling to use the media tree, in the same way we usually 
> > > virtually all other trees in KernelCI. The following commands gives 
> > > KernelCI a fatal error.
> > >
> > > git clone 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > > cd linux
> > > git remote add media https://git.linuxtv.org/media_tree.git
> > > git remote update media
> > > git fetch --tags https://git.linuxtv.org/media_tree.git
> > > fatal: Couldn't find remote ref HEAD  
> 
> First of all, the best is to use, instead, media_stage.git, as this
> contains the most recent patches. You should notice, however, that
> we might need to rebase media_stage.git. That doesn't happen often,
> but if you use it for CI, just in case, be sure that it will do a
> 
> 	git reset --hard FETCH_HEAD
> 
> To avoid potential problems if a rebase is needed.
> 
> We periodically update media_tree.git as well (which never rebases),
> but the update is manual. I usually do it once per week between
> -rc1 to -rc7, by the time I handle myself pull requests.
> 
> For CI, the best is to use media_stage.git.
> 
> -
> 
> With regards to merge/fetch, please use the git protocol
> whenever possible, e. g.:
> 
> 	git://linuxtv.org/media_tree.git
> 	git://linuxtv.org/media_stage.git
> 
> you may, alternatively use:
> 
> 	https://linuxtv.org/git/media_tree.git
> 	https://linuxtv.org/git/media_stage.git
> 
> The 4 above url's are handled by git daemon.
> 
> If you use, instead:
> 
> 	https://git.linuxtv.org/media_tree.git
> 
> It will use cgit, which will try to do the right thing, but
> results may not be consistent, as the main goal for cgit is to
> provide a web visual interface (which is IMO a *lot* better than
> what git??b provides).
> 
> The problem with https is that it may require a hook during
> push time to update https references:
> 
> 	git update-server-info

I actually changed the setup today, removing the need of having
such hook, and dropping it from all repositories there. Now,
the https server (either via cgit or directly) shouldn't need it
anymore.

I suspect that the problem of having HEAD outdated/broken from time
to time should be gone now, and all 3 alternatives should work
properly.

Please let me know if you find any issues.

Regards,
Mauro

Re: linux-media checkout failing for KernelCI

[Denys Fedoryshchenko]

On Fri, 2024-05-17 at 12:57 -0400, Gustavo Padovan wrote:
> 
> 
> On 5/17/24 11:29, Guenter Roeck wrote:
> > On Fri, May 17, 2024 at 6:41 AM Gustavo Padovan <gus@collabora.com>
> > wrote:
> > > 
> > > Hi Media maintainers,
> > > 
> > > We are struggling to use the media tree, in the same way we
> > > usually
> > > virtually all other trees in KernelCI. The following commands
> > > gives
> > > KernelCI a fatal error.
> > > 
> > > git clone
> > > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > > cd linux
> > > git remote add media https://git.linuxtv.org/media_tree.git
> > > git remote update media
> > > git fetch --tags https://git.linuxtv.org/media_tree.git
> > Why not "git fetch --tags media" instead of the last two commands ?
> > That is working for me, and it is what I always use when fetching
> > remote repositories.
> 
> Hmm. Maybe we were just being newbies with the way KernelCI does it.
> :)
Actually, the problem is that this "magical sequence" in the legacy
code was written years ago, and git blame gives incorrect results, so i
can't ask author "why?". 
It’s a sort of "Chesterton's Fence" in the legacy library.