* [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception
@ 2015-12-03 11:37 P J P
2015-12-03 12:11 ` Gerd Hoffmann
0 siblings, 1 reply; 5+ messages in thread
From: P J P @ 2015-12-03 11:37 UTC (permalink / raw)
To: qemu-devel; +Cc: Gerd Hoffmann, Lian Yihan
Hello Gerd,
A floating point exception issue in the VNC server PNG compression support was
reported by Mr Lian Yihan, CC'd here. Given below is a proposed (tested)patch
to fix this issue.
===
>From 1ca4818333d39fed6567e316e37f6a6516f59c69 Mon Sep 17 00:00:00 2001
From: Prasad J Pandit <pjp@fedoraproject.org>
Date: Thu, 3 Dec 2015 16:04:51 +0530
Subject: [PATCH] ui: vnc: avoid floating point exception
While sending 'SetPixelFormat' messages to a VNC server,
the client could set the 'red-max', 'green-max' and 'blue-max'
values to be zero. This leads to a floating point exception in
write_png_palette while doing frame buffer updates.
Reported-by: Lian Yihan <lianyihan@360.cn>
Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>
---
ui/vnc.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/ui/vnc.c b/ui/vnc.c
index caf82f5..52c6809 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2189,15 +2189,15 @@ static void set_pixel_format(VncState *vs,
return;
}
- vs->client_pf.rmax = red_max;
+ vs->client_pf.rmax = red_max ? red_max : 0xFF;
vs->client_pf.rbits = hweight_long(red_max);
vs->client_pf.rshift = red_shift;
vs->client_pf.rmask = red_max << red_shift;
- vs->client_pf.gmax = green_max;
+ vs->client_pf.gmax = green_max ? green_max : 0xFF;
vs->client_pf.gbits = hweight_long(green_max);
vs->client_pf.gshift = green_shift;
vs->client_pf.gmask = green_max << green_shift;
- vs->client_pf.bmax = blue_max;
+ vs->client_pf.bmax = blue_max ? blue_max : 0xFF;
vs->client_pf.bbits = hweight_long(blue_max);
vs->client_pf.bshift = blue_shift;
vs->client_pf.bmask = blue_max << blue_shift;
--
2.4.3
===
Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
47AF CE69 3A90 54AA 9045 1053 DD13 3D32 FE5B 041F
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception
2015-12-03 11:37 [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception P J P
@ 2015-12-03 12:11 ` Gerd Hoffmann
2015-12-03 12:32 ` Peter Maydell
0 siblings, 1 reply; 5+ messages in thread
From: Gerd Hoffmann @ 2015-12-03 12:11 UTC (permalink / raw)
To: P J P, Peter Maydell; +Cc: qemu-devel, Lian Yihan
On Do, 2015-12-03 at 17:07 +0530, P J P wrote:
> Hello Gerd,
>
> A floating point exception issue in the VNC server PNG compression support was
> reported by Mr Lian Yihan, CC'd here. Given below is a proposed (tested)patch
> to fix this issue.
> - vs->client_pf.rmax = red_max;
> + vs->client_pf.rmax = red_max ? red_max : 0xFF;
Looks good.
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
Peter, can you pick this up directly for 2.5-rc3?
thanks,
Gerd
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception
2015-12-03 12:11 ` Gerd Hoffmann
@ 2015-12-03 12:32 ` Peter Maydell
2015-12-03 13:24 ` P J P
0 siblings, 1 reply; 5+ messages in thread
From: Peter Maydell @ 2015-12-03 12:32 UTC (permalink / raw)
To: Gerd Hoffmann; +Cc: Lian Yihan, QEMU Developers, P J P
On 3 December 2015 at 12:11, Gerd Hoffmann <kraxel@redhat.com> wrote:
> On Do, 2015-12-03 at 17:07 +0530, P J P wrote:
>> Hello Gerd,
>>
>> A floating point exception issue in the VNC server PNG compression support was
>> reported by Mr Lian Yihan, CC'd here. Given below is a proposed (tested)patch
>> to fix this issue.
>
>> - vs->client_pf.rmax = red_max;
>> + vs->client_pf.rmax = red_max ? red_max : 0xFF;
>
> Looks good.
>
> Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
>
> Peter, can you pick this up directly for 2.5-rc3?
The patch doesn't apply to master. Can you produce a version
that does, please?
thanks
-- PMM
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception
2015-12-03 12:32 ` Peter Maydell
@ 2015-12-03 13:24 ` P J P
2015-12-03 13:57 ` Peter Maydell
0 siblings, 1 reply; 5+ messages in thread
From: P J P @ 2015-12-03 13:24 UTC (permalink / raw)
To: Peter Maydell; +Cc: Lian Yihan, Gerd Hoffmann, QEMU Developers
Hello Peter,
+-- On Thu, 3 Dec 2015, Peter Maydell wrote --+
| The patch doesn't apply to master. Can you produce a version
| that does, please?
Please see this new one, hope it works.
===
>From d4661b8d99f8c8439167d85165439c619553b933 Mon Sep 17 00:00:00 2001
From: Prasad J Pandit <pjp@fedoraproject.org>
Date: Thu, 3 Dec 2015 18:47:46 +0530
Subject: [PATCH] ui: vnc: avoid floating point exception
While sending 'SetPixelFormat' messages to a VNC server,
the client could set the 'red-max', 'green-max' and 'blue-max'
values to be zero. This leads to a floating point exception in
write_png_palette while doing frame buffer updates.
Reported-by: Lian Yihan <lianyihan@360.cn>
Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>
---
ui/vnc.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/ui/vnc.c b/ui/vnc.c
index 7538405..cbe4d33 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2198,15 +2198,15 @@ static void set_pixel_format(VncState *vs,
return;
}
- vs->client_pf.rmax = red_max;
+ vs->client_pf.rmax = red_max ? red_max : 0xFF;
vs->client_pf.rbits = hweight_long(red_max);
vs->client_pf.rshift = red_shift;
vs->client_pf.rmask = red_max << red_shift;
- vs->client_pf.gmax = green_max;
+ vs->client_pf.gmax = green_max ? green_max : 0xFF;
vs->client_pf.gbits = hweight_long(green_max);
vs->client_pf.gshift = green_shift;
vs->client_pf.gmask = green_max << green_shift;
- vs->client_pf.bmax = blue_max;
+ vs->client_pf.bmax = blue_max ? blue_max : 0xFF;
vs->client_pf.bbits = hweight_long(blue_max);
vs->client_pf.bshift = blue_shift;
vs->client_pf.bmask = blue_max << blue_shift;
--
2.4.3
===
Thank you.
--
Prasad J Pandit / Red Hat Product Security Team
47AF CE69 3A90 54AA 9045 1053 DD13 3D32 FE5B 041F
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception
2015-12-03 13:24 ` P J P
@ 2015-12-03 13:57 ` Peter Maydell
0 siblings, 0 replies; 5+ messages in thread
From: Peter Maydell @ 2015-12-03 13:57 UTC (permalink / raw)
To: P J P; +Cc: Lian Yihan, Gerd Hoffmann, QEMU Developers
On 3 December 2015 at 13:24, P J P <ppandit@redhat.com> wrote:
> Hello Peter,
>
> +-- On Thu, 3 Dec 2015, Peter Maydell wrote --+
> | The patch doesn't apply to master. Can you produce a version
> | that does, please?
>
> Please see this new one, hope it works.
>
> ===
> From d4661b8d99f8c8439167d85165439c619553b933 Mon Sep 17 00:00:00 2001
> From: Prasad J Pandit <pjp@fedoraproject.org>
> Date: Thu, 3 Dec 2015 18:47:46 +0530
> Subject: [PATCH] ui: vnc: avoid floating point exception
>
> While sending 'SetPixelFormat' messages to a VNC server,
> the client could set the 'red-max', 'green-max' and 'blue-max'
> values to be zero. This leads to a floating point exception in
> write_png_palette while doing frame buffer updates.
>
> Reported-by: Lian Yihan <lianyihan@360.cn>
> Signed-off-by: Prasad J Pandit <pjp@fedoraproject.org>
> ---
> ui/vnc.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
Applied, thanks.
-- PMM
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-12-03 13:57 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-03 11:37 [Qemu-devel] [PATCH] ui: vnc: avoid floating point exception P J P
2015-12-03 12:11 ` Gerd Hoffmann
2015-12-03 12:32 ` Peter Maydell
2015-12-03 13:24 ` P J P
2015-12-03 13:57 ` Peter Maydell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).