* [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
@ 2012-01-23 18:38 Jan Kiszka
2012-01-23 19:16 ` Stefan Weil
2012-01-24 8:39 ` Paolo Bonzini
0 siblings, 2 replies; 12+ messages in thread
From: Jan Kiszka @ 2012-01-23 18:38 UTC (permalink / raw)
To: qemu-devel, Anthony Liguori; +Cc: Stefan Weil, Christoph Egger, Alexander Graf
Forking an expr process for every byte of the input data slows down the
checksum calculation massively. Fix this while still remaining portable
by implementing the algorithm in awk.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
That "remaining portable" is an unproven claim. So please check that
problematic NetBSD and also mingw. Thanks!
scripts/signrom.sh | 18 ++++++++----------
1 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/scripts/signrom.sh b/scripts/signrom.sh
index 9dc5c63..f0f460e 100755
--- a/scripts/signrom.sh
+++ b/scripts/signrom.sh
@@ -23,22 +23,20 @@
# did we get proper arguments?
test "$1" -a "$2" || exit 1
-sum=0
-
# find out the file size
x=`dd if="$1" bs=1 count=1 skip=2 2>/dev/null | od -t u1 -A n`
-#size=`expr $x \* 512 - 1`
size=$(( $x * 512 - 1 ))
# now get the checksum
nums=`od -A n -t u1 -v -N $size "$1"`
-for i in ${nums}; do
- # add each byte's value to sum
- sum=`expr \( $sum + $i \) % 256`
-done
-
-sum=$(( (256 - $sum) % 256 ))
-sum_octal=$( printf "%o" $sum )
+sum_octal=`echo $nums | awk 'BEGIN {
+ getline data_str;
+ sum = 0;
+ n = split(data_str, data, " ");
+ for (i = 1; i <= n; i++)
+ sum = ( sum + data[i] ) % 256;
+ printf "%o", (256 - sum) % 256;
+}'`
# and write the output file
cp "$1" "$2"
--
1.7.3.4
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 18:38 [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation Jan Kiszka
@ 2012-01-23 19:16 ` Stefan Weil
2012-01-23 19:19 ` Jan Kiszka
2012-01-24 10:18 ` Christoph Egger
2012-01-24 8:39 ` Paolo Bonzini
1 sibling, 2 replies; 12+ messages in thread
From: Stefan Weil @ 2012-01-23 19:16 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Christoph Egger, Anthony Liguori, qemu-devel, Alexander Graf
Am 23.01.2012 19:38, schrieb Jan Kiszka:
> Forking an expr process for every byte of the input data slows down the
> checksum calculation massively. Fix this while still remaining portable
> by implementing the algorithm in awk.
>
> Signed-off-by: Jan Kiszka<jan.kiszka@siemens.com>
> ---
>
> That "remaining portable" is an unproven claim. So please check that
> problematic NetBSD and also mingw. Thanks!
>
> scripts/signrom.sh | 18 ++++++++----------
> 1 files changed, 8 insertions(+), 10 deletions(-)
>
> diff --git a/scripts/signrom.sh b/scripts/signrom.sh
> index 9dc5c63..f0f460e 100755
> --- a/scripts/signrom.sh
> +++ b/scripts/signrom.sh
> @@ -23,22 +23,20 @@
> # did we get proper arguments?
> test "$1" -a "$2" || exit 1
>
> -sum=0
> -
> # find out the file size
> x=`dd if="$1" bs=1 count=1 skip=2 2>/dev/null | od -t u1 -A n`
> -#size=`expr $x \* 512 - 1`
> size=$(( $x * 512 - 1 ))
>
> # now get the checksum
> nums=`od -A n -t u1 -v -N $size "$1"`
> -for i in ${nums}; do
> - # add each byte's value to sum
> - sum=`expr \( $sum + $i \) % 256`
> -done
> -
> -sum=$(( (256 - $sum) % 256 ))
> -sum_octal=$( printf "%o" $sum )
> +sum_octal=`echo $nums | awk 'BEGIN {
> + getline data_str;
> + sum = 0;
> + n = split(data_str, data, " ");
> + for (i = 1; i<= n; i++)
> + sum = ( sum + data[i] ) % 256;
> + printf "%o", (256 - sum) % 256;
> +}'`
>
> # and write the output file
> cp "$1" "$2"
>
What about replacing the whole script by a python script?
That would save about 6 more forks :-)
I'd prefer if we could get rid of all AWK dependencies in QEMU
and focus on as few scripting languages as reasonable.
Regards,
Stefan W.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 19:16 ` Stefan Weil
@ 2012-01-23 19:19 ` Jan Kiszka
2012-01-23 19:25 ` Stefan Weil
2012-01-24 10:18 ` Christoph Egger
1 sibling, 1 reply; 12+ messages in thread
From: Jan Kiszka @ 2012-01-23 19:19 UTC (permalink / raw)
To: Stefan Weil; +Cc: Christoph Egger, Anthony Liguori, qemu-devel, Alexander Graf
On 2012-01-23 20:16, Stefan Weil wrote:
> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>> Forking an expr process for every byte of the input data slows down the
>> checksum calculation massively. Fix this while still remaining portable
>> by implementing the algorithm in awk.
>>
>> Signed-off-by: Jan Kiszka<jan.kiszka@siemens.com>
>> ---
>>
>> That "remaining portable" is an unproven claim. So please check that
>> problematic NetBSD and also mingw. Thanks!
>>
>> scripts/signrom.sh | 18 ++++++++----------
>> 1 files changed, 8 insertions(+), 10 deletions(-)
>>
>> diff --git a/scripts/signrom.sh b/scripts/signrom.sh
>> index 9dc5c63..f0f460e 100755
>> --- a/scripts/signrom.sh
>> +++ b/scripts/signrom.sh
>> @@ -23,22 +23,20 @@
>> # did we get proper arguments?
>> test "$1" -a "$2" || exit 1
>>
>> -sum=0
>> -
>> # find out the file size
>> x=`dd if="$1" bs=1 count=1 skip=2 2>/dev/null | od -t u1 -A n`
>> -#size=`expr $x \* 512 - 1`
>> size=$(( $x * 512 - 1 ))
>>
>> # now get the checksum
>> nums=`od -A n -t u1 -v -N $size "$1"`
>> -for i in ${nums}; do
>> - # add each byte's value to sum
>> - sum=`expr \( $sum + $i \) % 256`
>> -done
>> -
>> -sum=$(( (256 - $sum) % 256 ))
>> -sum_octal=$( printf "%o" $sum )
>> +sum_octal=`echo $nums | awk 'BEGIN {
>> + getline data_str;
>> + sum = 0;
>> + n = split(data_str, data, " ");
>> + for (i = 1; i<= n; i++)
>> + sum = ( sum + data[i] ) % 256;
>> + printf "%o", (256 - sum) % 256;
>> +}'`
>>
>> # and write the output file
>> cp "$1" "$2"
>>
>
> What about replacing the whole script by a python script?
> That would save about 6 more forks :-)
I think we have no python dependency in the build system yet - while we
have for awk. That was also the original reason to go for a shell script.
If the situation changed, I would not vote against python, for sure.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 19:19 ` Jan Kiszka
@ 2012-01-23 19:25 ` Stefan Weil
2012-01-23 20:08 ` Anthony Liguori
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Weil @ 2012-01-23 19:25 UTC (permalink / raw)
To: Jan Kiszka; +Cc: qemu-devel
Am 23.01.2012 20:19, schrieb Jan Kiszka:
> On 2012-01-23 20:16, Stefan Weil wrote:
>> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>>
[snip]
>> What about replacing the whole script by a python script?
>> That would save about 6 more forks :-)
>
> I think we have no python dependency in the build system yet - while we
> have for awk. That was also the original reason to go for a shell script.
>
> If the situation changed, I would not vote against python, for sure.
>
> Jan
We do. Even configure will fail now without Python:
if ! has $python; then
echo "Python not found. Use --python=/path/to/python"
exit 1
fi
QEMU depends on Python = Python 2 for QAPI, tracing and maybe more.
See scripts/*.py.
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 19:25 ` Stefan Weil
@ 2012-01-23 20:08 ` Anthony Liguori
2012-01-23 20:32 ` Jan Kiszka
2012-01-24 8:18 ` Paolo Bonzini
0 siblings, 2 replies; 12+ messages in thread
From: Anthony Liguori @ 2012-01-23 20:08 UTC (permalink / raw)
To: Stefan Weil; +Cc: Jan Kiszka, qemu-devel
On 01/23/2012 01:25 PM, Stefan Weil wrote:
> Am 23.01.2012 20:19, schrieb Jan Kiszka:
>> On 2012-01-23 20:16, Stefan Weil wrote:
>>> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>>>
> [snip]
>>> What about replacing the whole script by a python script?
>>> That would save about 6 more forks :-)
>>
>> I think we have no python dependency in the build system yet - while we
>> have for awk. That was also the original reason to go for a shell script.
>>
>> If the situation changed, I would not vote against python, for sure.
>>
>> Jan
>
> We do. Even configure will fail now without Python:
>
> if ! has $python; then
> echo "Python not found. Use --python=/path/to/python"
> exit 1
> fi
>
> QEMU depends on Python = Python 2 for QAPI, tracing and maybe more.
> See scripts/*.py.
I've been thinking that we could potentially rewrite a large chunk (all?) of
configure in python too since we have such a hard dependency now.
Regards,
Anthony Liguori
>
> Stefan
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 20:08 ` Anthony Liguori
@ 2012-01-23 20:32 ` Jan Kiszka
2012-01-23 20:37 ` Anthony Liguori
2012-01-24 8:18 ` Paolo Bonzini
1 sibling, 1 reply; 12+ messages in thread
From: Jan Kiszka @ 2012-01-23 20:32 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Stefan Weil, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]
On 2012-01-23 21:08, Anthony Liguori wrote:
> On 01/23/2012 01:25 PM, Stefan Weil wrote:
>> Am 23.01.2012 20:19, schrieb Jan Kiszka:
>>> On 2012-01-23 20:16, Stefan Weil wrote:
>>>> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>>>>
>> [snip]
>>>> What about replacing the whole script by a python script?
>>>> That would save about 6 more forks :-)
>>>
>>> I think we have no python dependency in the build system yet - while we
>>> have for awk. That was also the original reason to go for a shell
>>> script.
>>>
>>> If the situation changed, I would not vote against python, for sure.
>>>
>>> Jan
>>
>> We do. Even configure will fail now without Python:
>>
>> if ! has $python; then
>> echo "Python not found. Use --python=/path/to/python"
>> exit 1
>> fi
>>
>> QEMU depends on Python = Python 2 for QAPI, tracing and maybe more.
>> See scripts/*.py.
>
> I've been thinking that we could potentially rewrite a large chunk
> (all?) of configure in python too since we have such a hard dependency now.
Sounds almost like a GSoC project. Or work for someone who did something
nasty ;). At least it's no afternoon hack when done properly.
However, I will now write signrom.py. That's much shorter.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 20:32 ` Jan Kiszka
@ 2012-01-23 20:37 ` Anthony Liguori
2012-01-23 21:32 ` Stefan Weil
2012-01-24 9:43 ` Andreas Färber
0 siblings, 2 replies; 12+ messages in thread
From: Anthony Liguori @ 2012-01-23 20:37 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Stefan Weil, qemu-devel
On 01/23/2012 02:32 PM, Jan Kiszka wrote:
> On 2012-01-23 21:08, Anthony Liguori wrote:
>> On 01/23/2012 01:25 PM, Stefan Weil wrote:
>>> Am 23.01.2012 20:19, schrieb Jan Kiszka:
>>>> On 2012-01-23 20:16, Stefan Weil wrote:
>>>>> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>>>>>
>>> [snip]
>>>>> What about replacing the whole script by a python script?
>>>>> That would save about 6 more forks :-)
>>>>
>>>> I think we have no python dependency in the build system yet - while we
>>>> have for awk. That was also the original reason to go for a shell
>>>> script.
>>>>
>>>> If the situation changed, I would not vote against python, for sure.
>>>>
>>>> Jan
>>>
>>> We do. Even configure will fail now without Python:
>>>
>>> if ! has $python; then
>>> echo "Python not found. Use --python=/path/to/python"
>>> exit 1
>>> fi
>>>
>>> QEMU depends on Python = Python 2 for QAPI, tracing and maybe more.
>>> See scripts/*.py.
>>
>> I've been thinking that we could potentially rewrite a large chunk
>> (all?) of configure in python too since we have such a hard dependency now.
>
> Sounds almost like a GSoC project.
That's a good idea!
> Or work for someone who did something nasty ;).
Shhh, don't tell any students that :-)
Regards,
Anthony Liguori
> At least it's no afternoon hack when done properly.
>
> However, I will now write signrom.py. That's much shorter.
>
> Jan
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 20:37 ` Anthony Liguori
@ 2012-01-23 21:32 ` Stefan Weil
2012-01-24 9:43 ` Andreas Färber
1 sibling, 0 replies; 12+ messages in thread
From: Stefan Weil @ 2012-01-23 21:32 UTC (permalink / raw)
To: Jan Kiszka; +Cc: qemu-devel
On 01/23/2012 02:32 PM, Jan Kiszka wrote:
> On 2012-01-23 21:08, Anthony Liguori wrote:
>>
>> I've been thinking that we could potentially rewrite a large chunk
>> (all?) of configure in python too since we have such a hard
>> dependency now.
[snip]
> However, I will now write signrom.py. That's much shorter.
>
> Jan
I think that will improve that script (and maybe configure later) a lot.
Let me just add a small remark:
We require python2 now, but please try to be compatible with python3
as far as possible. It _is_ possible, at least to some degree
(I tried it with the existing QEMU scripts).
I don't believe that QEMU will stick to python2 forever, so avoiding
unnecessary incompatibilities now will save us time later.
Cheers,
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 20:37 ` Anthony Liguori
2012-01-23 21:32 ` Stefan Weil
@ 2012-01-24 9:43 ` Andreas Färber
1 sibling, 0 replies; 12+ messages in thread
From: Andreas Färber @ 2012-01-24 9:43 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Stefan Weil, Jan Kiszka, qemu-devel
Am 23.01.2012 21:37, schrieb Anthony Liguori:
> On 01/23/2012 02:32 PM, Jan Kiszka wrote:
>> On 2012-01-23 21:08, Anthony Liguori wrote:
>>> I've been thinking that we could potentially rewrite a large chunk
>>> (all?) of configure in python too since we have such a hard
>>> dependency now.
>>
>> Sounds almost like a GSoC project.
>
> That's a good idea!
Having a signrom.py, driven by configure or make, is fine.
But configure should be able to run anywhere and at least report what's
missing. #!/bin/sh is the standard, highly compatible mechanism among
Unices, Posices and more to do just that. Python on the other hand is
either /usr/bin/python or /usr/bin/env python, which both don't work on
systems without /usr/bin for starters.
So please not all of configure, at least detect and/or allow to
overwrite the python path.
Andreas
--
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 20:08 ` Anthony Liguori
2012-01-23 20:32 ` Jan Kiszka
@ 2012-01-24 8:18 ` Paolo Bonzini
1 sibling, 0 replies; 12+ messages in thread
From: Paolo Bonzini @ 2012-01-24 8:18 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Stefan Weil, qemu-devel, Jan Kiszka
On 01/23/2012 09:08 PM, Anthony Liguori wrote:
>>
>> QEMU depends on Python = Python 2 for QAPI, tracing and maybe more.
>> See scripts/*.py.
>
> I've been thinking that we could potentially rewrite a large chunk
> (all?) of configure in python too since we have such a hard dependency now.
What would the advantages be??
Paolo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 19:16 ` Stefan Weil
2012-01-23 19:19 ` Jan Kiszka
@ 2012-01-24 10:18 ` Christoph Egger
1 sibling, 0 replies; 12+ messages in thread
From: Christoph Egger @ 2012-01-24 10:18 UTC (permalink / raw)
To: Stefan Weil; +Cc: Jan Kiszka, Anthony Liguori, qemu-devel, Alexander Graf
On 01/23/12 20:16, Stefan Weil wrote:
> Am 23.01.2012 19:38, schrieb Jan Kiszka:
>> Forking an expr process for every byte of the input data slows down the
>> checksum calculation massively. Fix this while still remaining portable
>> by implementing the algorithm in awk.
>>
>> Signed-off-by: Jan Kiszka<jan.kiszka@siemens.com>
>> ---
>>
>> That "remaining portable" is an unproven claim. So please check that
>> problematic NetBSD and also mingw. Thanks!
The "unproven claim" is actually what SuSv3 spec
(http://pubs.opengroup.org/onlinepubs/009695399/utilities/od.html)
says:
"and the output from each of the transformations shall be separated by
one or more <blank>s."
The actual problem is that the programmer expects the behaviour
of either GNU od(1) or AT&T UNIX od(1) rather the SuSv3 spec.
NetBSD has AT&T UNIX od(1).
Christoph
>>
>> scripts/signrom.sh | 18 ++++++++----------
>> 1 files changed, 8 insertions(+), 10 deletions(-)
>>
>> diff --git a/scripts/signrom.sh b/scripts/signrom.sh
>> index 9dc5c63..f0f460e 100755
>> --- a/scripts/signrom.sh
>> +++ b/scripts/signrom.sh
>> @@ -23,22 +23,20 @@
>> # did we get proper arguments?
>> test "$1" -a "$2" || exit 1
>>
>> -sum=0
>> -
>> # find out the file size
>> x=`dd if="$1" bs=1 count=1 skip=2 2>/dev/null | od -t u1 -A n`
>> -#size=`expr $x \* 512 - 1`
>> size=$(( $x * 512 - 1 ))
>>
>> # now get the checksum
>> nums=`od -A n -t u1 -v -N $size "$1"`
>> -for i in ${nums}; do
>> - # add each byte's value to sum
>> - sum=`expr \( $sum + $i \) % 256`
>> -done
>> -
>> -sum=$(( (256 - $sum) % 256 ))
>> -sum_octal=$( printf "%o" $sum )
>> +sum_octal=`echo $nums | awk 'BEGIN {
>> + getline data_str;
>> + sum = 0;
>> + n = split(data_str, data, " ");
>> + for (i = 1; i<= n; i++)
>> + sum = ( sum + data[i] ) % 256;
>> + printf "%o", (256 - sum) % 256;
>> +}'`
>>
>> # and write the output file
>> cp "$1" "$2"
>
> What about replacing the whole script by a python script?
> That would save about 6 more forks :-)
>
> I'd prefer if we could get rid of all AWK dependencies in QEMU
> and focus on as few scripting languages as reasonable.
>
> Regards,
> Stefan W.
>
>
--
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation
2012-01-23 18:38 [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation Jan Kiszka
2012-01-23 19:16 ` Stefan Weil
@ 2012-01-24 8:39 ` Paolo Bonzini
1 sibling, 0 replies; 12+ messages in thread
From: Paolo Bonzini @ 2012-01-24 8:39 UTC (permalink / raw)
To: Jan Kiszka
Cc: Stefan Weil, Anthony Liguori, Christoph Egger, qemu-devel,
Alexander Graf
On 01/23/2012 07:38 PM, Jan Kiszka wrote:
> nums=`od -A n -t u1 -v -N $size "$1"`
> -for i in ${nums}; do
> - # add each byte's value to sum
> - sum=`expr \( $sum + $i \) % 256`
> -done
> -
> -sum=$(( (256 - $sum) % 256 ))
We have a bashism here, so why not do
sum=$(( ($sum + $i) % 256 ))
and do the same with a one-line patch (two line if you change the header
to #!/bin/bash)? It should be just as fast as awk.
Paolo
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2012-01-24 11:04 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-23 18:38 [Qemu-devel] [RFC][PATCH] signrom: Speed up checksum calculation Jan Kiszka
2012-01-23 19:16 ` Stefan Weil
2012-01-23 19:19 ` Jan Kiszka
2012-01-23 19:25 ` Stefan Weil
2012-01-23 20:08 ` Anthony Liguori
2012-01-23 20:32 ` Jan Kiszka
2012-01-23 20:37 ` Anthony Liguori
2012-01-23 21:32 ` Stefan Weil
2012-01-24 9:43 ` Andreas Färber
2012-01-24 8:18 ` Paolo Bonzini
2012-01-24 10:18 ` Christoph Egger
2012-01-24 8:39 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).