[FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

Hello,

So it would be great if the filter-branch beside the Rewrite
f8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also
append a basic ETA signaling the end of the operation.

It could be as simple as the the average number of milliseconds per
step up to this point multiplied with the remaining number of steps,
then convert this into a day:hour:minutes:seconds format. It sound
simple enough, but really handy for long running filter branch
operations. I could also contribute if one could direct me towards the
appropriate files this should go to.

Thanks,

Bernát GÁBOR

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Jeff King]

On Tue, Aug 25, 2015 at 05:01:01PM +0200, Gabor Bernat wrote:

> So it would be great if the filter-branch beside the Rewrite
> f8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also
> append a basic ETA signaling the end of the operation.
> 
> It could be as simple as the the average number of milliseconds per
> step up to this point multiplied with the remaining number of steps,
> then convert this into a day:hour:minutes:seconds format. It sound
> simple enough, but really handy for long running filter branch
> operations. I could also contribute if one could direct me towards the
> appropriate files this should go to.

Yeah, I agree the current filter-branch progress reporting is pretty
simplistic. The line you want to tweak is in git-filter-branch.sh:

  printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"

But the real trick is getting accurate timing in a shell script. You can
probably do the math within a "$(())" arithmetic block if you're OK with
integers. But you'd have to run `date` on each loop iteration to get the
current time, which may have a noticeable speed impact.

Of course, filter-branch is so slow in the first place, maybe it would
not matter. :)

Something like this seems to work:

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 5b3f63d..04e45bc 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -276,10 +276,21 @@ test $commits -eq 0 && die "Found nothing to rewrite"
 
 # Rewrite the commits
 
+start=$(date +%s)
 git_filter_branch__commit_count=0
 while read commit parents; do
 	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
-	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"
+	now=$(date +%s)
+	elapsed=$(($now - $start))
+	# work in integer percentages as a sort of fixed-point
+	pct=$(($git_filter_branch__commit_count * 100 / $commits))
+	if test $pct -eq 0; then
+		remain=
+	else
+		eta=$(($elapsed * 100 / $pct))
+		remain="($(($eta - $elapsed)) seconds remaining)   "
+	fi
+	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain"
 
 	case "$filter_subdir" in
 	"")

but the time jumps around early on because of the lack of precision. And
of course there's no smoothing, and no emphasis on recent history versus
the whole operation. I'll leave those as an exercise to the reader. :)

-Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> +start=$(date +%s)

Is that a GNU extension?

>  git_filter_branch__commit_count=0
>  while read commit parents; do
>  	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
> -	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"
> +	now=$(date +%s)
> +	elapsed=$(($now - $start))
> +	# work in integer percentages as a sort of fixed-point
> +	pct=$(($git_filter_branch__commit_count * 100 / $commits))
> +	if test $pct -eq 0; then
> +		remain=
> +	else
> +		eta=$(($elapsed * 100 / $pct))
> +		remain="($(($eta - $elapsed)) seconds remaining)   "
> +	fi
> +	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain"
>  
>  	case "$filter_subdir" in
>  	"")
>
> but the time jumps around early on because of the lack of precision. And
> of course there's no smoothing, and no emphasis on recent history versus
> the whole operation. I'll leave those as an exercise to the reader. :)

;-)

An alternative implementation may be to ask `date` every 1000
commits (or whatever sufficiently large value that we can amortise
the cost) to measure the rate and compute $remain based on that
measurement.  That way, we can afford to use more portable ways to
ask `date` about the current time and compute the "how many seconds"
ourselves.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Jeff King]

On Tue, Aug 25, 2015 at 11:33:49AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > +start=$(date +%s)
> 
> Is that a GNU extension?

Thanks, I meant to mention that, too. POSIX has "+" formats, but
apparently no way to get an integer number of seconds. I don't know how
widely "%s" is supported; BSD "date" seems to know about it.

> An alternative implementation may be to ask `date` every 1000
> commits (or whatever sufficiently large value that we can amortise
> the cost) to measure the rate and compute $remain based on that
> measurement.  That way, we can afford to use more portable ways to
> ask `date` about the current time and compute the "how many seconds"
> ourselves.

Yeah, that would probably be a good solution, assuming there is a
portable "how many seconds" (I do not relish the thought of
reconstructing it based on the current hours/minutes/seconds).

I wonder how awful it would be to make a tool like "git-progress", where
you'd tell it "--total=$commits --eta" on the command line, and then
occasionally print the current count its stdin. It might be a little
painful to use, though. You'd want to background it with a pipe to its
stdin, which is annoying without bash-style "<()" anonymous pipes.

-Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Jeff King]

On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:

> Yeah, that would probably be a good solution, assuming there is a
> portable "how many seconds" (I do not relish the thought of
> reconstructing it based on the current hours/minutes/seconds).

A little googling came up with:

    awk 'END { print systime() }' </dev/null

which probably (?) works everywhere. I feel dirty just having typed
that, though.

-Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

I would lean for an extra on-demand flag for this, and a per commit
measurement, initial noise is okay for the first iteration I think.
Secondly note that on the output other messages could also be present
(other than the rewrite), as the command running may have its own
output. I will try to create a initial version just have some time for
it :)

On 8/25/15, Jeff King <peff@peff.net> wrote:
> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:
>
>> Yeah, that would probably be a good solution, assuming there is a
>> portable "how many seconds" (I do not relish the thought of
>> reconstructing it based on the current hours/minutes/seconds).
>
> A little googling came up with:
>
>     awk 'END { print systime() }' </dev/null
>
> which probably (?) works everywhere. I feel dirty just having typed
> that, though.
>
> -Peff
>


-- 
 Bernát Gábor
Student -  Budapest University of Technology and Economics - Computer
Engineering, M.Sc. - Budapest, Hungary
System Integrator - Gravity R&D | Rock Solid Recommendations - Budapest
office | 5-7 Expo ter | H-1101 | Budapest | Hungary

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

On Tue, Aug 25, 2015 at 2:54 PM, Jeff King <peff@peff.net> wrote:
> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:
>
>> Yeah, that would probably be a good solution, assuming there is a
>> portable "how many seconds" (I do not relish the thought of
>> reconstructing it based on the current hours/minutes/seconds).
>
> A little googling came up with:
>
>     awk 'END { print systime() }' </dev/null
>
> which probably (?) works everywhere.

On Mac OS X and FreeBSD:

    $ awk 'END { print systime() }' </dev/null
    awk: calling undefined function systime
    source line number 1
    $

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Jeff King]

On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:

> > A little googling came up with:
> >
> >     awk 'END { print systime() }' </dev/null
> >
> > which probably (?) works everywhere.
> 
> On Mac OS X and FreeBSD:
> 
>     $ awk 'END { print systime() }' </dev/null
>     awk: calling undefined function systime
>     source line number 1
>     $

Oh, well. The reference I saw was that the old Kernighan nawk had it,
but that seems not to be the case:

  http://www.cs.princeton.edu/~bwk/btl.mirror/

"date +%s" seems to work on OS X, and so presumably on other BSDs. No
clue what would work on stuff like SunOS, AIX, etc.

-Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

Hello,

Here's what I ended up using, and seemed to work well:
https://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0

Does this looks okay, should I create a patch from this?

Thanks,
Bernát GÁBOR


On Wed, Aug 26, 2015 at 4:15 AM, Jeff King <peff@peff.net> wrote:
> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:
>
>> > A little googling came up with:
>> >
>> >     awk 'END { print systime() }' </dev/null
>> >
>> > which probably (?) works everywhere.
>>
>> On Mac OS X and FreeBSD:
>>
>>     $ awk 'END { print systime() }' </dev/null
>>     awk: calling undefined function systime
>>     source line number 1
>>     $
>
> Oh, well. The reference I saw was that the old Kernighan nawk had it,
> but that seems not to be the case:
>
>   http://www.cs.princeton.edu/~bwk/btl.mirror/
>
> "date +%s" seems to work on OS X, and so presumably on other BSDs. No
> clue what would work on stuff like SunOS, AIX, etc.
>
> -Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
Bernát GÁBOR


On Sat, Aug 29, 2015 at 11:50 AM, Gabor Bernat <bernat@primeranks.net> wrote:
> Hello,
>
> Here's what I ended up using, and seemed to work well:
> https://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0
>
> Does this looks okay, should I create a patch from this?
>
> Thanks,
> Bernát GÁBOR
>
>
> On Wed, Aug 26, 2015 at 4:15 AM, Jeff King <peff@peff.net> wrote:
>> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:
>>
>>> > A little googling came up with:
>>> >
>>> >     awk 'END { print systime() }' </dev/null
>>> >
>>> > which probably (?) works everywhere.
>>>
>>> On Mac OS X and FreeBSD:
>>>
>>>     $ awk 'END { print systime() }' </dev/null
>>>     awk: calling undefined function systime
>>>     source line number 1
>>>     $
>>
>> Oh, well. The reference I saw was that the old Kernighan nawk had it,
>> but that seems not to be the case:
>>
>>   http://www.cs.princeton.edu/~bwk/btl.mirror/
>>
>> "date +%s" seems to work on OS X, and so presumably on other BSDs. No
>> clue what would work on stuff like SunOS, AIX, etc.
>>
>> -Peff

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
> Does this looks okay, should I create a patch from this?

Excerpt:

    now=$(date +%s)
    elapsed=$(($now - $start))
    remaining_second=$((...))
    eta=$(($now + $remaining_second))
    finish_by=$(date -d "@$eta")

Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets
the kernel's value for Daylight Saving Time, rather than displaying
the specified time as in Linux.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

Reading after it, I think the most close we can get with this is, awk
'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
this value (and avoid displaying it) if that fails too. Do you agree?
Bernát GÁBOR


On Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
>> Does this looks okay, should I create a patch from this?
>
> Excerpt:
>
>     now=$(date +%s)
>     elapsed=$(($now - $start))
>     remaining_second=$((...))
>     eta=$(($now + $remaining_second))
>     finish_by=$(date -d "@$eta")
>
> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets
> the kernel's value for Daylight Saving Time, rather than displaying
> the specified time as in Linux.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

(Please don't top-post on this list.)

On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> Reading after it, I think the most close we can get with this is, awk
> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
> this value (and avoid displaying it) if that fails too. Do you agree?

strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
FreeBSD, or even the default awk on Linux (which is mawk on Linux
installations I've checked).

Most portable likely would be Perl, however, that's probably too
heavyweight inside a loop like this, even if called only once each N
iterations.

> On Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
>>> Does this looks okay, should I create a patch from this?
>>
>> Excerpt:
>>
>>     now=$(date +%s)
>>     elapsed=$(($now - $start))
>>     remaining_second=$((...))
>>     eta=$(($now + $remaining_second))
>>     finish_by=$(date -d "@$eta")
>>
>> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets
>> the kernel's value for Daylight Saving Time, rather than displaying
>> the specified time as in Linux.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Mikael Magnusson]

On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> (Please don't top-post on this list.)
>
> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>> Reading after it, I think the most close we can get with this is, awk
>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>> this value (and avoid displaying it) if that fails too. Do you agree?
>
> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
> FreeBSD, or even the default awk on Linux (which is mawk on Linux
> installations I've checked).
>
> Most portable likely would be Perl, however, that's probably too
> heavyweight inside a loop like this, even if called only once each N
> iterations.

http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
Found this,

awk 'BEGIN{srand();print srand()}'

srand() in awk returns the previous seed value, and calling it without
an argument sets it to time of day, so the above sequence should
return seconds since the epoch, or at least something in seconds that
is relative to a fixed point which is all that's needed in this
thread.

-- 
Mikael Magnusson

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

this can work instead of the data command for getting the time
elapsed, however for getting the actual date of a timestamp is not
possible generally; so I think I will just remove that part.
Bernát GÁBOR


On Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> (Please don't top-post on this list.)
>>
>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Reading after it, I think the most close we can get with this is, awk
>>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>>> this value (and avoid displaying it) if that fails too. Do you agree?
>>
>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
>> FreeBSD, or even the default awk on Linux (which is mawk on Linux
>> installations I've checked).
>>
>> Most portable likely would be Perl, however, that's probably too
>> heavyweight inside a loop like this, even if called only once each N
>> iterations.
>
> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
> Found this,
>
> awk 'BEGIN{srand();print srand()}'
>
> srand() in awk returns the previous seed value, and calling it without
> an argument sets it to time of day, so the above sequence should
> return seconds since the epoch, or at least something in seconds that
> is relative to a fixed point which is all that's needed in this
> thread.
>
> --
> Mikael Magnusson

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

On Sun, Aug 30, 2015 at 4:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Reading after it, I think the most close we can get with this is, awk
>>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>>> this value (and avoid displaying it) if that fails too. Do you agree?
>>
>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
>> FreeBSD, or even the default awk on Linux (which is mawk on Linux
>> installations I've checked).
>>
>> Most portable likely would be Perl, however, that's probably too
>> heavyweight inside a loop like this, even if called only once each N
>> iterations.
>
> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
> Found this,
>
> awk 'BEGIN{srand();print srand()}'
>
> srand() in awk returns the previous seed value, and calling it without
> an argument sets it to time of day, so the above sequence should
> return seconds since the epoch, or at least something in seconds that
> is relative to a fixed point which is all that's needed in this
> thread.

Indeed, this seems to be portable in my tests, and presumably works on
Solaris, whereas "date +%s" doesn't (according to that stackoverflow
answer).

> this can work instead of the data command for getting the time
> elapsed, however for getting the actual date of a timestamp is not
> possible generally; so I think I will just remove that part.

Agreed. I was going to suggest the same.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Mikael Magnusson]

On Sun, Aug 30, 2015 at 10:11 AM, Gabor Bernat <bernat@primeranks.net> wrote:
> this can work instead of the data command for getting the time
> elapsed, however for getting the actual date of a timestamp is not
> possible generally; so I think I will just remove that part.
> Bernát GÁBOR
>
>
> On Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
>> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> (Please don't top-post on this list.)

Ah, I got caught up on the ETA part. Do note that date +%s is also a
gnu extension and won't work everywhere.

-- 
Mikael Magnusson

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

>>> Most portable likely would be Perl, however, that's probably too
>>> heavyweight inside a loop like this, even if called only once each N
>>> iterations.

I think that is true.  Now, when it is too heavy to spawn perl,
would it be light enough to spawn awk, I have to wonder.  Even if
the implementation uses awk, I think the time measurement should be
done only once each N iterations (e.g. every 1000 commits measure
the rate and divide the remaining commits with that rate while
displaying the progress; if you are chewing 100 commits per minute
and have 2000 commits to go, you know it will take 20 more minutes).

>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>> Found this,
>>
>> awk 'BEGIN{srand();print srand()}'
>>
>> srand() in awk returns the previous seed value, and calling it without
>> an argument sets it to time of day, so the above sequence should
>> return seconds since the epoch, or at least something in seconds that
>> is relative to a fixed point which is all that's needed in this
>> thread.

In practice this should work, but it makes me feel somewhat uneasy.

POSIX says "Set the seed value for rand to expr or use the time of
day if expr is omitted. The previous seed value shall be returned."
but I do not see anything that says that "the time of day" is
counted in seconds around there (which is the crucial bit for this
application).

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
(4.15 Seconds since the Epoch) says "The relationship between the
actual time of day and the current value for seconds since the Epoch
is unspecified."

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Gabor Bernat]

I would argue against the every n commit check, or at least making it
configurable, as in my case the speed is something between 0.01 and
1.5 seconds per commit. Checking it every n commit would make it I
feel quite slow to adapt. But it's debatable.

On 8/30/15, Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
>
>>>> Most portable likely would be Perl, however, that's probably too
>>>> heavyweight inside a loop like this, even if called only once each N
>>>> iterations.
>
> I think that is true.  Now, when it is too heavy to spawn perl,
> would it be light enough to spawn awk, I have to wonder.  Even if
> the implementation uses awk, I think the time measurement should be
> done only once each N iterations (e.g. every 1000 commits measure
> the rate and divide the remaining commits with that rate while
> displaying the progress; if you are chewing 100 commits per minute
> and have 2000 commits to go, you know it will take 20 more minutes).
>
>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>> Found this,
>>>
>>> awk 'BEGIN{srand();print srand()}'
>>>
>>> srand() in awk returns the previous seed value, and calling it without
>>> an argument sets it to time of day, so the above sequence should
>>> return seconds since the epoch, or at least something in seconds that
>>> is relative to a fixed point which is all that's needed in this
>>> thread.
>
> In practice this should work, but it makes me feel somewhat uneasy.
>
> POSIX says "Set the seed value for rand to expr or use the time of
> day if expr is omitted. The previous seed value shall be returned."
> but I do not see anything that says that "the time of day" is
> counted in seconds around there (which is the crucial bit for this
> application).
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
> (4.15 Seconds since the Epoch) says "The relationship between the
> actual time of day and the current value for seconds since the Epoch
> is unspecified."
>


-- 
 Bernát Gábor
Student -  Budapest University of Technology and Economics - Computer
Engineering, M.Sc. - Budapest, Hungary
System Integrator - Gravity R&D | Rock Solid Recommendations - Budapest
office | 5-7 Expo ter | H-1101 | Budapest | Hungary

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

On Sun, Aug 30, 2015 at 12:52 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>> Found this,
>>>
>>> awk 'BEGIN{srand();print srand()}'
>>>
>>> srand() in awk returns the previous seed value, and calling it without
>>> an argument sets it to time of day, so the above sequence should
>>> return seconds since the epoch, or at least something in seconds that
>>> is relative to a fixed point which is all that's needed in this
>>> thread.
>
> In practice this should work, but it makes me feel somewhat uneasy.
>
> POSIX says "Set the seed value for rand to expr or use the time of
> day if expr is omitted. The previous seed value shall be returned."
> but I do not see anything that says that "the time of day" is
> counted in seconds around there (which is the crucial bit for this
> application).
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
> (4.15 Seconds since the Epoch) says "The relationship between the
> actual time of day and the current value for seconds since the Epoch
> is unspecified."

I suppose a viable approach might be to test once outside the loop if
"date +%s" is supported and print the "(%d elapsed / %d estimated
remaining)" annotation within the loop if it is, else not. The test
might look something like this:

echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t

Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date
+%s" would get the annotated output, whereas it would fall back
gracefully to the non-annotated output on platforms such as Solaris
(and perhaps AIX) which lack it.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Mikael Magnusson]

On Sun, Aug 30, 2015 at 6:58 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> I would argue against the every n commit check, or at least making it
> configurable, as in my case the speed is something between 0.01 and
> 1.5 seconds per commit. Checking it every n commit would make it I
> feel quite slow to adapt. But it's debatable.



You must have missed the previous times someone said this, but please
don't top post on this list.



Here are some timings for running the command in question 1000 times
on my computer:
awk 'BEGIN{srand();print srand()}'
 0.32s user 1.20s system 65% cpu 2.332 total

perl -e 'print time'
 0.69s user 1.45s system 73% cpu 2.921 total

date +%s
 0.27s user 0.99s system 78% cpu 1.604 total

and for comparison,
/bin/true
 0.02s user 0.26s system 24% cpu 1.127 total

-- 
Mikael Magnusson

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

(please don't top-post on this list)

On Sun, Aug 30, 2015 at 12:58 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> I would argue against the every n commit check, or at least making it
> configurable, as in my case the speed is something between 0.01 and
> 1.5 seconds per commit. Checking it every n commit would make it I
> feel quite slow to adapt. But it's debatable.

I'm wondering why these two decisions ("showing estimated time" and
"frequency of the computation") should be put in the hands of the user
in the first place.

1. Why have a --progress-eta option at all as opposed to just enabling
it unconditionally if the platform can support it ("date +%s") and if
it can be done without impacting the overall runtime noticeably?

2. Why make the user responsible for deciding how often to do the time
check (via some configuration) as opposed to adjusting it dynamically
based upon how quickly the operation is proceeding. That is, if the
filter-branch operation is zipping along at 0.01 seconds per commit,
then the time check can be done less frequently (say every 50 or 100
commits) so that it doesn't slow the overall operation. Conversely, if
the operation is molasses, moving at 2 seconds per commit, then doing
the time check more frequently (perhaps after each commit) probably
won't noticeably impact the user's perception of the operation's
progress.


> On 8/30/15, Junio C Hamano <gitster@pobox.com> wrote:
>> Eric Sunshine <sunshine@sunshineco.com> writes:
>>
>>>>> Most portable likely would be Perl, however, that's probably too
>>>>> heavyweight inside a loop like this, even if called only once each N
>>>>> iterations.
>>
>> I think that is true.  Now, when it is too heavy to spawn perl,
>> would it be light enough to spawn awk, I have to wonder.  Even if
>> the implementation uses awk, I think the time measurement should be
>> done only once each N iterations (e.g. every 1000 commits measure
>> the rate and divide the remaining commits with that rate while
>> displaying the progress; if you are chewing 100 commits per minute
>> and have 2000 commits to go, you know it will take 20 more minutes).
>>
>>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>>> Found this,
>>>>
>>>> awk 'BEGIN{srand();print srand()}'
>>>>
>>>> srand() in awk returns the previous seed value, and calling it without
>>>> an argument sets it to time of day, so the above sequence should
>>>> return seconds since the epoch, or at least something in seconds that
>>>> is relative to a fixed point which is all that's needed in this
>>>> thread.
>>
>> In practice this should work, but it makes me feel somewhat uneasy.
>>
>> POSIX says "Set the seed value for rand to expr or use the time of
>> day if expr is omitted. The previous seed value shall be returned."
>> but I do not see anything that says that "the time of day" is
>> counted in seconds around there (which is the crucial bit for this
>> application).
>>
>> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
>> (4.15 Seconds since the Epoch) says "The relationship between the
>> actual time of day and the current value for seconds since the Epoch
>> is unspecified."

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Eric Sunshine]

On Sun, Aug 30, 2015 at 03:40:20PM -0400, Eric Sunshine wrote:
> I suppose a viable approach might be to test once outside the loop if
> "date +%s" is supported and print the "(%d elapsed / %d estimated
> remaining)" annotation within the loop if it is, else not. The test
> might look something like this:
> 
> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t

Actually, you'd also want to suppress 'date' errors via /dev/null, so
perhaps:

  { echo $(date +%s) | grep -q '^[0-9][0-9]*$'; } 2>/dev/null && show_eta=t

or something.

> Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date
> +%s" would get the annotated output, whereas it would fall back
> gracefully to the non-annotated output on platforms such as Solaris
> (and perhaps AIX) which lack it.

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> I suppose a viable approach might be to test once outside the loop if
> "date +%s" is supported and print the "(%d elapsed / %d estimated
> remaining)" annotation within the loop if it is, else not. The test
> might look something like this:
>
> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t
>
> Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date
> +%s" would get the annotated output, whereas it would fall back
> gracefully to the non-annotated output on platforms such as Solaris
> (and perhaps AIX) which lack it.

I kind of like this for its simplicity, especially given that it
would cover the vast majority of users in practice.