[Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

The new git-submodule.sh script writes .git-submodule-status to
the source directory every time no matter what. This makes it conditional.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

I compile out of tree on a remote guest system where I mount the
source directory as "readonly" and build directory as "rw" and
scripts/git-submodule.sh tries writing to the source directory even when
I manually update modules on a host machine which is quite annoying.

Is this something acceptable? Or I am missing something here?

---
 scripts/git-submodule.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
index d8fbc7e47e..b642994a67 100755
--- a/scripts/git-submodule.sh
+++ b/scripts/git-submodule.sh
@@ -33,6 +33,8 @@ status)
     ;;
 update)
     git submodule update --init $modules 1>/dev/null 2>&1
-    git submodule status $modules > "${substat}"
+    substat_tmp=$(mktemp)
+    git submodule status $modules > "$substat_tmp"
+    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
     ;;
 esac
-- 
2.11.0

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Daniel P. Berrange]

On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
> The new git-submodule.sh script writes .git-submodule-status to
> the source directory every time no matter what. This makes it conditional.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> 
> I compile out of tree on a remote guest system where I mount the
> source directory as "readonly" and build directory as "rw" and
> scripts/git-submodule.sh tries writing to the source directory even when
> I manually update modules on a host machine which is quite annoying.
> 
> Is this something acceptable? Or I am missing something here?

How did you update the modules - did you manually run  'git submodule update...'
or did you use the git-submodule.sh script on your host machine ?

If you run git-submodule.sh on the host, then it should save the status
file, and then when you run make on the guest system, it should notice
that you're already updated and never even invoke 'git-submodule.sh update'

I'm not against your proposal below, but I'm curious why you're seeing
'git-submodule.sh update' being run by make in the first place.

> 
> ---
>  scripts/git-submodule.sh | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
> index d8fbc7e47e..b642994a67 100755
> --- a/scripts/git-submodule.sh
> +++ b/scripts/git-submodule.sh
> @@ -33,6 +33,8 @@ status)
>      ;;
>  update)
>      git submodule update --init $modules 1>/dev/null 2>&1
> -    git submodule status $modules > "${substat}"
> +    substat_tmp=$(mktemp)
> +    git submodule status $modules > "$substat_tmp"
> +    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
>      ;;
>  esac
> -- 
> 2.11.0
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Peter Maydell]

On 24 October 2017 at 17:27, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>> I compile out of tree on a remote guest system where I mount the
>> source directory as "readonly" and build directory as "rw" and
>> scripts/git-submodule.sh tries writing to the source directory even when
>> I manually update modules on a host machine which is quite annoying.
>>
>> Is this something acceptable? Or I am missing something here?
>
> How did you update the modules - did you manually run  'git submodule update...'
> or did you use the git-submodule.sh script on your host machine ?
>
> If you run git-submodule.sh on the host, then it should save the status
> file, and then when you run make on the guest system, it should notice
> that you're already updated and never even invoke 'git-submodule.sh update'
>
> I'm not against your proposal below, but I'm curious why you're seeing
> 'git-submodule.sh update' being run by make in the first place.

I don't think "git checkout on the fileserver, then build on the
remote machine" is a particularly weird workflow. I'm starting
to feel that the idea of doing git updates during "make" is not
so good as it initially seemed; it's just not something that
people expect to have happen during the build step.

thanks
-- PMM

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

On 25/10/17 03:33, Peter Maydell wrote:
> On 24 October 2017 at 17:27, Daniel P. Berrange <berrange@redhat.com> wrote:
>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>>> I compile out of tree on a remote guest system where I mount the
>>> source directory as "readonly" and build directory as "rw" and
>>> scripts/git-submodule.sh tries writing to the source directory even when
>>> I manually update modules on a host machine which is quite annoying.
>>>
>>> Is this something acceptable? Or I am missing something here?
>>
>> How did you update the modules - did you manually run  'git submodule update...'
>> or did you use the git-submodule.sh script on your host machine ?
>>
>> If you run git-submodule.sh on the host, then it should save the status
>> file, and then when you run make on the guest system, it should notice
>> that you're already updated and never even invoke 'git-submodule.sh update'
>>
>> I'm not against your proposal below, but I'm curious why you're seeing
>> 'git-submodule.sh update' being run by make in the first place.
> 
> I don't think "git checkout on the fileserver, then build on the
> remote machine" is a particularly weird workflow. I'm starting
> to feel that the idea of doing git updates during "make" is not
> so good as it initially seemed; it's just not something that
> people expect to have happen during the build step.


This is exactly the point. When you do out of tree compile, the last thing
you want is "make" to touch anything in the source tree.


-- 
Alexey

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

On 25/10/17 03:27, Daniel P. Berrange wrote:
> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>> The new git-submodule.sh script writes .git-submodule-status to
>> the source directory every time no matter what. This makes it conditional.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>
>> I compile out of tree on a remote guest system where I mount the
>> source directory as "readonly" and build directory as "rw" and
>> scripts/git-submodule.sh tries writing to the source directory even when
>> I manually update modules on a host machine which is quite annoying.
>>
>> Is this something acceptable? Or I am missing something here?
> 
> How did you update the modules - did you manually run  'git submodule update...'
> or did you use the git-submodule.sh script on your host machine ?


I run scripts/git-submodule.sh. Which is not thrilling either as I rather
expect source tree not to be affected in any way when running "make".


> If you run git-submodule.sh on the host, then it should save the status
> file, and then when you run make on the guest system, it should notice
> that you're already updated and never even invoke 'git-submodule.sh update'


scripts/git-submodule.sh also tries writing to the source directory (I
should probably have fixed that branch too) but this failure is not fatal
for "make" but makes it want to try "update" and then "make" fails.


> I'm not against your proposal below, but I'm curious why you're seeing
> 'git-submodule.sh update' being run by make in the first place.
> 
>>
>> ---
>>  scripts/git-submodule.sh | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
>> index d8fbc7e47e..b642994a67 100755
>> --- a/scripts/git-submodule.sh
>> +++ b/scripts/git-submodule.sh
>> @@ -33,6 +33,8 @@ status)
>>      ;;
>>  update)
>>      git submodule update --init $modules 1>/dev/null 2>&1
>> -    git submodule status $modules > "${substat}"
>> +    substat_tmp=$(mktemp)
>> +    git submodule status $modules > "$substat_tmp"
>> +    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
>>      ;;
>>  esac
>> -- 
>> 2.11.0
>>
> 
> Regards,
> Daniel
> 


-- 
Alexey

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Daniel P. Berrange]

On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
> On 25/10/17 03:27, Daniel P. Berrange wrote:
> > On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
> >> The new git-submodule.sh script writes .git-submodule-status to
> >> the source directory every time no matter what. This makes it conditional.
> >>
> >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> >> ---
> >>
> >> I compile out of tree on a remote guest system where I mount the
> >> source directory as "readonly" and build directory as "rw" and
> >> scripts/git-submodule.sh tries writing to the source directory even when
> >> I manually update modules on a host machine which is quite annoying.
> >>
> >> Is this something acceptable? Or I am missing something here?
> > 
> > How did you update the modules - did you manually run  'git submodule update...'
> > or did you use the git-submodule.sh script on your host machine ?
> 
> 
> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
> expect source tree not to be affected in any way when running "make".

Oh, did you pass the list of sub-modules to it when running

eg, ./scripts/git-submodule.sh update ui/keycodemapdb

the list of submodules you need is printed in the configure output summary.

> > If you run git-submodule.sh on the host, then it should save the status
> > file, and then when you run make on the guest system, it should notice
> > that you're already updated and never even invoke 'git-submodule.sh update'
> 
> 
> scripts/git-submodule.sh also tries writing to the source directory (I
> should probably have fixed that branch too) but this failure is not fatal
> for "make" but makes it want to try "update" and then "make" fails.

This shouldn't have happened in your case though, if you have already run
'git-submodule.sh update ...list of modules...' on the host machine, with
the same list of modules that the guest 'configure' printed out.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

On 25/10/17 17:57, Daniel P. Berrange wrote:
> On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
>> On 25/10/17 03:27, Daniel P. Berrange wrote:
>>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>>>> The new git-submodule.sh script writes .git-submodule-status to
>>>> the source directory every time no matter what. This makes it conditional.
>>>>
>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>> ---
>>>>
>>>> I compile out of tree on a remote guest system where I mount the
>>>> source directory as "readonly" and build directory as "rw" and
>>>> scripts/git-submodule.sh tries writing to the source directory even when
>>>> I manually update modules on a host machine which is quite annoying.
>>>>
>>>> Is this something acceptable? Or I am missing something here?
>>>
>>> How did you update the modules - did you manually run  'git submodule update...'
>>> or did you use the git-submodule.sh script on your host machine ?
>>
>>
>> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
>> expect source tree not to be affected in any way when running "make".
> 
> Oh, did you pass the list of sub-modules to it when running
> 
> eg, ./scripts/git-submodule.sh update ui/keycodemapdb
> 
> the list of submodules you need is printed in the configure output summary.

Sure, otherwise it does nothing.


> 
>>> If you run git-submodule.sh on the host, then it should save the status
>>> file, and then when you run make on the guest system, it should notice
>>> that you're already updated and never even invoke 'git-submodule.sh update'
>>
>>
>> scripts/git-submodule.sh also tries writing to the source directory (I
>> should probably have fixed that branch too) but this failure is not fatal
>> for "make" but makes it want to try "update" and then "make" fails.
> 
> This shouldn't have happened in your case though, if you have already run
> 'git-submodule.sh update ...list of modules...' on the host machine, with
> the same list of modules that the guest 'configure' printed out.

It does not matter if I run git-submodule.sh or not - "git-submodule.sh
status" will try writing to the read only folder anyway and it will fail
and Makefile's  git_module_status will be set to 1.

If I do as below (and that's what I should have done as I said), then
"git-submodule.sh update" is not invoked and we are good. I am not
reposting it yet as 1) my shell skills are crap (need to delete the temp
file or rewrite the whole thing not to use temp file or rewrite it in
python - why do not people use python everywhere?!) 2) I still hope we stop
doing this from Makefile :)



diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
index d8fbc7e47e..ced00b6cae 100755
--- a/scripts/git-submodule.sh
+++ b/scripts/git-submodule.sh
@@ -23,16 +23,20 @@ then
     exit 1
 fi

+substat_tmp=$(mktemp)
+
 case "$command" in
 status)
     test -f "$substat" || exit 1
-    trap "rm -f ${substat}.tmp" EXIT
-    git submodule status $modules > "${substat}.tmp"
-    diff "${substat}" "${substat}.tmp" >/dev/null
+    git submodule status $modules > "$substat_tmp"
+    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
+    ( rm "${substat_tmp}" 2>/dev/null )
     exit $?
     ;;
 update)
     git submodule update --init $modules 1>/dev/null 2>&1
-    git submodule status $modules > "${substat}"
+    git submodule status $modules > "$substat_tmp"
+    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
+    ( rm "${substat_tmp}" 2>/dev/null )
     ;;
 esac



-- 
Alexey

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Daniel P. Berrange]

On Wed, Oct 25, 2017 at 07:10:40PM +1100, Alexey Kardashevskiy wrote:
> On 25/10/17 17:57, Daniel P. Berrange wrote:
> > On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
> >> On 25/10/17 03:27, Daniel P. Berrange wrote:
> >>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
> >>>> The new git-submodule.sh script writes .git-submodule-status to
> >>>> the source directory every time no matter what. This makes it conditional.
> >>>>
> >>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> >>>> ---
> >>>>
> >>>> I compile out of tree on a remote guest system where I mount the
> >>>> source directory as "readonly" and build directory as "rw" and
> >>>> scripts/git-submodule.sh tries writing to the source directory even when
> >>>> I manually update modules on a host machine which is quite annoying.
> >>>>
> >>>> Is this something acceptable? Or I am missing something here?
> >>>
> >>> How did you update the modules - did you manually run  'git submodule update...'
> >>> or did you use the git-submodule.sh script on your host machine ?
> >>
> >>
> >> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
> >> expect source tree not to be affected in any way when running "make".
> > 
> > Oh, did you pass the list of sub-modules to it when running
> > 
> > eg, ./scripts/git-submodule.sh update ui/keycodemapdb
> > 
> > the list of submodules you need is printed in the configure output summary.
> 
> Sure, otherwise it does nothing.
> 
> 
> > 
> >>> If you run git-submodule.sh on the host, then it should save the status
> >>> file, and then when you run make on the guest system, it should notice
> >>> that you're already updated and never even invoke 'git-submodule.sh update'
> >>
> >>
> >> scripts/git-submodule.sh also tries writing to the source directory (I
> >> should probably have fixed that branch too) but this failure is not fatal
> >> for "make" but makes it want to try "update" and then "make" fails.
> > 
> > This shouldn't have happened in your case though, if you have already run
> > 'git-submodule.sh update ...list of modules...' on the host machine, with
> > the same list of modules that the guest 'configure' printed out.
> 
> It does not matter if I run git-submodule.sh or not - "git-submodule.sh
> status" will try writing to the read only folder anyway and it will fail
> and Makefile's  git_module_status will be set to 1.


Ahhhh, great, now I understand why you're hitting the problem !

> If I do as below (and that's what I should have done as I said), then
> "git-submodule.sh update" is not invoked and we are good. I am not
> reposting it yet as 1) my shell skills are crap (need to delete the temp
> file or rewrite the whole thing not to use temp file or rewrite it in
> python - why do not people use python everywhere?!) 2) I still hope we stop
> doing this from Makefile :)

I agree using a tmpfile is the right fix here.

> 
> 
> 
> diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
> index d8fbc7e47e..ced00b6cae 100755
> --- a/scripts/git-submodule.sh
> +++ b/scripts/git-submodule.sh
> @@ -23,16 +23,20 @@ then
>      exit 1
>  fi
> 
> +substat_tmp=$(mktemp)
> +
>  case "$command" in
>  status)
>      test -f "$substat" || exit 1
> -    trap "rm -f ${substat}.tmp" EXIT
> -    git submodule status $modules > "${substat}.tmp"
> -    diff "${substat}" "${substat}.tmp" >/dev/null
> +    git submodule status $modules > "$substat_tmp"
> +    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"

We shouldn't be overwriting the existing status file during 'status' - only
'update' should overwrite. 'status' should be a no-op query only.

> +    ( rm "${substat_tmp}" 2>/dev/null )
>      exit $?
>      ;;
>  update)
>      git submodule update --init $modules 1>/dev/null 2>&1
> -    git submodule status $modules > "${substat}"
> +    git submodule status $modules > "$substat_tmp"
> +    diff "${substat_tmp}" "${substat}" || mv "${substat_tmp}" "${substat}"
> +    ( rm "${substat_tmp}" 2>/dev/null )
>      ;;
>  esac
> 
> 
> 
> -- 
> Alexey

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

On 26/10/17 08:11, Daniel P. Berrange wrote:
> On Wed, Oct 25, 2017 at 07:10:40PM +1100, Alexey Kardashevskiy wrote:
>> On 25/10/17 17:57, Daniel P. Berrange wrote:
>>> On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
>>>> On 25/10/17 03:27, Daniel P. Berrange wrote:
>>>>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>>>>>> The new git-submodule.sh script writes .git-submodule-status to
>>>>>> the source directory every time no matter what. This makes it conditional.
>>>>>>
>>>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>>>> ---
>>>>>>
>>>>>> I compile out of tree on a remote guest system where I mount the
>>>>>> source directory as "readonly" and build directory as "rw" and
>>>>>> scripts/git-submodule.sh tries writing to the source directory even when
>>>>>> I manually update modules on a host machine which is quite annoying.
>>>>>>
>>>>>> Is this something acceptable? Or I am missing something here?
>>>>>
>>>>> How did you update the modules - did you manually run  'git submodule update...'
>>>>> or did you use the git-submodule.sh script on your host machine ?
>>>>
>>>>
>>>> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
>>>> expect source tree not to be affected in any way when running "make".
>>>
>>> Oh, did you pass the list of sub-modules to it when running
>>>
>>> eg, ./scripts/git-submodule.sh update ui/keycodemapdb
>>>
>>> the list of submodules you need is printed in the configure output summary.
>>
>> Sure, otherwise it does nothing.
>>
>>
>>>
>>>>> If you run git-submodule.sh on the host, then it should save the status
>>>>> file, and then when you run make on the guest system, it should notice
>>>>> that you're already updated and never even invoke 'git-submodule.sh update'
>>>>
>>>>
>>>> scripts/git-submodule.sh also tries writing to the source directory (I
>>>> should probably have fixed that branch too) but this failure is not fatal
>>>> for "make" but makes it want to try "update" and then "make" fails.
>>>
>>> This shouldn't have happened in your case though, if you have already run
>>> 'git-submodule.sh update ...list of modules...' on the host machine, with
>>> the same list of modules that the guest 'configure' printed out.
>>
>> It does not matter if I run git-submodule.sh or not - "git-submodule.sh
>> status" will try writing to the read only folder anyway and it will fail
>> and Makefile's  git_module_status will be set to 1.
> 
> 
> Ahhhh, great, now I understand why you're hitting the problem !
> 
>> If I do as below (and that's what I should have done as I said), then
>> "git-submodule.sh update" is not invoked and we are good. I am not
>> reposting it yet as 1) my shell skills are crap (need to delete the temp
>> file or rewrite the whole thing not to use temp file or rewrite it in
>> python - why do not people use python everywhere?!) 2) I still hope we stop
>> doing this from Makefile :)
> 
> I agree using a tmpfile is the right fix here.


I still think not doing "git update" from Makefile is the right fix here,
is that a final decision? Why cannot "configure" do this (and ideally have
a way not to do this at all, like --no-git-submodules-update)?  Just
checking...



-- 
Alexey

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Daniel P. Berrange]

On Thu, Oct 26, 2017 at 11:54:59AM +1100, Alexey Kardashevskiy wrote:
> On 26/10/17 08:11, Daniel P. Berrange wrote:
> > On Wed, Oct 25, 2017 at 07:10:40PM +1100, Alexey Kardashevskiy wrote:
> >> On 25/10/17 17:57, Daniel P. Berrange wrote:
> >>> On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
> >>>> On 25/10/17 03:27, Daniel P. Berrange wrote:
> >>>>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
> >>>>>> The new git-submodule.sh script writes .git-submodule-status to
> >>>>>> the source directory every time no matter what. This makes it conditional.
> >>>>>>
> >>>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> >>>>>> ---
> >>>>>>
> >>>>>> I compile out of tree on a remote guest system where I mount the
> >>>>>> source directory as "readonly" and build directory as "rw" and
> >>>>>> scripts/git-submodule.sh tries writing to the source directory even when
> >>>>>> I manually update modules on a host machine which is quite annoying.
> >>>>>>
> >>>>>> Is this something acceptable? Or I am missing something here?
> >>>>>
> >>>>> How did you update the modules - did you manually run  'git submodule update...'
> >>>>> or did you use the git-submodule.sh script on your host machine ?
> >>>>
> >>>>
> >>>> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
> >>>> expect source tree not to be affected in any way when running "make".
> >>>
> >>> Oh, did you pass the list of sub-modules to it when running
> >>>
> >>> eg, ./scripts/git-submodule.sh update ui/keycodemapdb
> >>>
> >>> the list of submodules you need is printed in the configure output summary.
> >>
> >> Sure, otherwise it does nothing.
> >>
> >>
> >>>
> >>>>> If you run git-submodule.sh on the host, then it should save the status
> >>>>> file, and then when you run make on the guest system, it should notice
> >>>>> that you're already updated and never even invoke 'git-submodule.sh update'
> >>>>
> >>>>
> >>>> scripts/git-submodule.sh also tries writing to the source directory (I
> >>>> should probably have fixed that branch too) but this failure is not fatal
> >>>> for "make" but makes it want to try "update" and then "make" fails.
> >>>
> >>> This shouldn't have happened in your case though, if you have already run
> >>> 'git-submodule.sh update ...list of modules...' on the host machine, with
> >>> the same list of modules that the guest 'configure' printed out.
> >>
> >> It does not matter if I run git-submodule.sh or not - "git-submodule.sh
> >> status" will try writing to the read only folder anyway and it will fail
> >> and Makefile's  git_module_status will be set to 1.
> > 
> > 
> > Ahhhh, great, now I understand why you're hitting the problem !
> > 
> >> If I do as below (and that's what I should have done as I said), then
> >> "git-submodule.sh update" is not invoked and we are good. I am not
> >> reposting it yet as 1) my shell skills are crap (need to delete the temp
> >> file or rewrite the whole thing not to use temp file or rewrite it in
> >> python - why do not people use python everywhere?!) 2) I still hope we stop
> >> doing this from Makefile :)
> > 
> > I agree using a tmpfile is the right fix here.
> 
> 
> I still think not doing "git update" from Makefile is the right fix here,
> is that a final decision? Why cannot "configure" do this (and ideally have
> a way not to do this at all, like --no-git-submodules-update)?  Just
> checking...

That ends up doing exactly the same thing. We would need to make sure that
configure gets re-run when the submodules change. So you just go from having
'make' run git-submodule.sh, to have make run config.status which runs
configure, which runs git-submodule.sh. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Alexey Kardashevskiy]

On 26/10/17 18:02, Daniel P. Berrange wrote:
> On Thu, Oct 26, 2017 at 11:54:59AM +1100, Alexey Kardashevskiy wrote:
>> On 26/10/17 08:11, Daniel P. Berrange wrote:
>>> On Wed, Oct 25, 2017 at 07:10:40PM +1100, Alexey Kardashevskiy wrote:
>>>> On 25/10/17 17:57, Daniel P. Berrange wrote:
>>>>> On Wed, Oct 25, 2017 at 12:45:10PM +1100, Alexey Kardashevskiy wrote:
>>>>>> On 25/10/17 03:27, Daniel P. Berrange wrote:
>>>>>>> On Tue, Oct 24, 2017 at 07:58:53PM +1100, Alexey Kardashevskiy wrote:
>>>>>>>> The new git-submodule.sh script writes .git-submodule-status to
>>>>>>>> the source directory every time no matter what. This makes it conditional.
>>>>>>>>
>>>>>>>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>>>>>>>> ---
>>>>>>>>
>>>>>>>> I compile out of tree on a remote guest system where I mount the
>>>>>>>> source directory as "readonly" and build directory as "rw" and
>>>>>>>> scripts/git-submodule.sh tries writing to the source directory even when
>>>>>>>> I manually update modules on a host machine which is quite annoying.
>>>>>>>>
>>>>>>>> Is this something acceptable? Or I am missing something here?
>>>>>>>
>>>>>>> How did you update the modules - did you manually run  'git submodule update...'
>>>>>>> or did you use the git-submodule.sh script on your host machine ?
>>>>>>
>>>>>>
>>>>>> I run scripts/git-submodule.sh. Which is not thrilling either as I rather
>>>>>> expect source tree not to be affected in any way when running "make".
>>>>>
>>>>> Oh, did you pass the list of sub-modules to it when running
>>>>>
>>>>> eg, ./scripts/git-submodule.sh update ui/keycodemapdb
>>>>>
>>>>> the list of submodules you need is printed in the configure output summary.
>>>>
>>>> Sure, otherwise it does nothing.
>>>>
>>>>
>>>>>
>>>>>>> If you run git-submodule.sh on the host, then it should save the status
>>>>>>> file, and then when you run make on the guest system, it should notice
>>>>>>> that you're already updated and never even invoke 'git-submodule.sh update'
>>>>>>
>>>>>>
>>>>>> scripts/git-submodule.sh also tries writing to the source directory (I
>>>>>> should probably have fixed that branch too) but this failure is not fatal
>>>>>> for "make" but makes it want to try "update" and then "make" fails.
>>>>>
>>>>> This shouldn't have happened in your case though, if you have already run
>>>>> 'git-submodule.sh update ...list of modules...' on the host machine, with
>>>>> the same list of modules that the guest 'configure' printed out.
>>>>
>>>> It does not matter if I run git-submodule.sh or not - "git-submodule.sh
>>>> status" will try writing to the read only folder anyway and it will fail
>>>> and Makefile's  git_module_status will be set to 1.
>>>
>>>
>>> Ahhhh, great, now I understand why you're hitting the problem !
>>>
>>>> If I do as below (and that's what I should have done as I said), then
>>>> "git-submodule.sh update" is not invoked and we are good. I am not
>>>> reposting it yet as 1) my shell skills are crap (need to delete the temp
>>>> file or rewrite the whole thing not to use temp file or rewrite it in
>>>> python - why do not people use python everywhere?!) 2) I still hope we stop
>>>> doing this from Makefile :)
>>>
>>> I agree using a tmpfile is the right fix here.
>>
>>
>> I still think not doing "git update" from Makefile is the right fix here,
>> is that a final decision? Why cannot "configure" do this (and ideally have
>> a way not to do this at all, like --no-git-submodules-update)?  Just
>> checking...
> 
> That ends up doing exactly the same thing. We would need to make sure that
> configure gets re-run when the submodules change.


Submodules change once/twice a year so we could live quite happily without
this automation imho. What if I do not want submodules to be updated in
order to reproduce some bug and confirm that the newer version of submodule
has fixed it?

btw I looked now to my tree with upstream QEMU and dtc in that tree is 8
month old (v1.4.4 and there is v1.4.5 available). Is not it supposed to
update to something more recent automatically? git-submodule.sh does not do
checkout or pull.


> So you just go from having
> 'make' run git-submodule.sh, to have make run config.status which runs
> configure, which runs git-submodule.sh. 





-- 
Alexey

Re: [Qemu-devel] [RFC PATCH qemu] git-submodule.sh: Do not try writing to source directory if not necessary

[Peter Maydell]

On 26 October 2017 at 08:45, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> Submodules change once/twice a year so we could live quite happily without
> this automation imho.

This is a significant underestimate. The openbios submodule alone
has been updated 12 times this year.

> What if I do not want submodules to be updated in
> order to reproduce some bug and confirm that the newer version of submodule
> has fixed it?

The only thing we support is "use the version of the submodule that
goes with the git commit of QEMU that they're supposed to go with".
The only reason we have this automation at all is that git submodule
is a broken design that doesn't update the submodule automatically
on git checkout/git branch etc.

I'm not a huge fan of this automation as we have it at the moment, but
we do need to do something just to work around the terrible terrible
stock git behaviour :-(

> btw I looked now to my tree with upstream QEMU and dtc in that tree is 8
> month old (v1.4.4 and there is v1.4.5 available). Is not it supposed to
> update to something more recent automatically? git-submodule.sh does not do
> checkout or pull.

We update the submodules when we need to update them (which for
dependencies like dtc is usually "when we have some requirement
that mandates a newer version".)

thanks
-- PMM